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Executive  Summary 


Helix,  a  project  of  the  Systems  Engineering  Research  Center  (SERC),  is  a  multi-year  longitudinal  study 
designed  to  build  an  understanding  of  the  landscape  of  the  systems  engineering  workforce,  what 
enables  and  inhibits  the  effectiveness  of  systems  engineers,  and  how  organizations  are  attempting  to 
improve  their  effectiveness. 

Helix  is  exploring  three  research  questions: 

•  What  are  the  characteristics  of  systems  engineers? 

•  How  effective  are  systems  engineers  and  why? 

•  What  are  employers  doing  to  improve  the  effectiveness  of  their  systems  engineers? 

Note  that  Helix  is  not  studying  "systems  engineering"  per  se.  Helix  is  focused  instead  on  the  people  who 
perform  systems  engineering.  The  distinction  is  important  and  permeates  how  the  research  is 
conducted. 

In  its  first  year.  Helix  obtained  information  from  seven  organizations  within  the  US  Department  of 
Defense  (DoD)  and  the  US  defense  industrial  base  (DIB).  That  information  came  from  a  combination  of 
institutional  reports  (policies,  demographics,  career  paths,  etc.)  and  90-minute  interviews  with  112 
individuals,  including  systems  engineers  and  those  who  work  with  systems  engineers. 

To  date,  the  Helix  team  offers  the  following  insights  on  the  three  research  questions  above: 

1.  Question  1  -  What  are  the  Characteristics  of  Systems  Engineers? 

Among  the  most  important  findings  from  the  initial  interviews  is  clarity  about  the  mix  of  personal 
characteristics  and  technical  competencies  that  are  most  important  in  a  strong  systems  engineer. 

Strong  characteristics  in  leadership,  communications,  big  picture  thinking,  personal  networking, 
organization,  being  comfortable  with  uncertainty,  understanding  details,  balancing  conflicting 
information,  and  knowing  the  right  questions  to  ask  are  common  characteristics  of  the  best  systems 
engineers.  Several  of  these  characteristics  are  in  tension  -  e.g.,  big  picture  thinking  and 
understanding  details.  A  good  systems  engineer  can  balance  these  conflicting  characteristics. 

The  most  important  technical  competencies  include  a  solid  understanding  of  least  one  discipline  such 
as  mechanical  or  electrical  engineering,  the  ability  to  apply  that  understanding  in  depth  within  a 
domain  such  as  telecommunications,  and  the  ability  to  understand  other  engineering  disciplines  to 
the  point  of  being  able  to  ask  insightful  questions.  Besides  helping  a  systems  engineer  make  the  right 
technical  decisions,  these  competencies  increase  the  system  engineer's  credibility  as  a  technical 
leader. 

Three  systems  engineering-specific  competencies  stand  out:  being  able  to  apply  the  overall  systems 
engineering  process,  being  able  to  work  with  stakeholders  to  develop  clear,  testable  requirements, 
and  understanding  how  all  the  system  pieces  fit  together. 

Typically  there  is  a  transition  from  an  engineer  being  a  "specialty  engineer"  to  becoming  a  "systems 
engineer",  and  this  can  happen  at  any  point  in  the  career  path.  In  this  transition,  it  is  important  for 
the  individual  to  focus  on  breadth  rather  than  depth.  Some  disciplinary  experts  fail  to  make  this 
transition  well,  which  can  make  them  ineffective  as  systems  engineers. 
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2.  Question  2  -  How  effective  are  systems  engineers  and  why? 

Understanding  the  effectiveness  of  systems  engineers  is  complicated.  Generally,  interviewees  had 
difficulty  articulating  what  it  means  for  them  to  be  effective  beyond  staying  within  cost,  schedule, 
and  required  performance  parameters.  Many  said  they  were  effective  if  they  were  able  to  follow 
organizational  processes  and  procedures.  Others  said  being  effective  means  identifying  issues  and 
anticipating  risks.  Most  interviewees  built  on  their  views  regarding  important  characteristics  and 
competencies;  e.g.,  many  said  they  were  effective  if  they  were  able  to  pull  together  the  right  people 
on  their  team  and  have  them  work  in  harmony  -  a  direct  consequence  of  their  leadership, 
communications,  and  personal  networking  characteristics.  Many  who  were  interviewed  emphasized 
being  able  to  predict  and  articulate  the  long-range  impacts  of  actions.  The  significant  time  lag 
between  a  systems  engineer's  decision  and  others  being  able  to  see  the  results  of  that  decision  can 
hinder  the  ability  of  others  to  understand  and  appreciate  the  important  role  systems  engineers  play. 

Interviewees  articulated  several  important  benefits  they  offer,  including:  translating  highly  technical 
information  from  subject  matter  experts  (SMEs)  into  common  language  that  other  stakeholders  can 
understand;  eliciting  the  right  requirements  from  the  customer;  balancing  traditional  project 
management  cost  and  schedule  concerns  with  technical  requirements;  seeing  relationships  between 
engineering  disciplines  and  asking  the  right  questions  to  balance  these  disciplines;  and  managing 
changes  in  requirements;  and  understanding  future  implications  of  current  decisions. 

Interviewees  were  asked  what  most  improves  their  effectiveness.  The  two  most  common  responses 
were  strong  mentoring  and  diverse  experiences.  Conversely,  the  most  commonly  cited  inhibitors  to 
effectiveness  were  confusion  within  the  organization  as  to  what  systems  engineering  is  and  who  the 
systems  engineers  are,  the  long  time  lag  between  when  systems  engineers  make  decisions  and  when 
the  effects  of  those  decisions  are  apparent,  a  failure  of  non-systems  engineers  to  understand  and 
appreciate  the  value  that  systems  engineers  bring,  and  compressed  schedules. 

3.  Question  3  -  What  are  employers  doing  to  improve  their  systems  engineers'  effectiveness? 

Although  the  Helix  team  has  gained  some  insight  into  initiatives  being  conducted  to  improve  systems 
engineers,  it  is  premature  to  report  these  findings  here.  First  findings  on  employer  initiatives  will  be 
described  in  the  next  report. 

In  2014,  the  Helix  team  will  continue  to  analyze  data  collected  to  date  and  enrich  that  data  with  more 
interviews,  institutional  data,  and  perhaps  other  types  of  data  sources  as  well.  The  next  report  will  be 
published  in  Spring  2014.  Additionally,  Helix  will  hold  one  or  two  workshops  in  2014  to  explore  and 
refine  its  findings  with  experts  across  the  community;  a  primary  goal  of  these  workshops  will  be 
facilitating  organizational  understanding  of  the  findings,  helping  organizations  understand  where  these 
insights  may  provide  keys  to  improving  their  own  workforce  initiatives,  and  helping  organizations  tailor 
their  workforce  initiatives  based  on  these  insights. 
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1.  Introduction 


The  US  Department  of  Defense  (DoD)  and  the  Defense  Industrial  Base  (DIB),  comprising  contractors  that 
develop  and  deliver  systems  to  the  DoD,  have  been  facing  major  systems  engineering  challenges  in 
recent  years  (e.g.  GAO  2008,  2011,  2012).  Mission  requirements  are  evolving  and  they  demand  ever 
more  sophisticated  and  complex  systems  (e.g.  Boehm  et  al.  2010;  INCOSE  Technical  Operations  2007; 
Davidz  and  Nightingale  2007,  Frank  et  al.  2007);  the  tools,  processes,  and  technologies  that  systems 
engineers  must  master  keep  changing  ever  more  rapidly  (e.g.  Frank  2006);  and  budgets  and  schedules 
are  being  compressed  dramatically.  Certainly,  one  of  the  more  significant  concerns  is  that  thousands  of 
systems  engineers  in  the  defense  workforce  are  nearing  retirement  and  will  be  removing  with  them 
hundreds  of  thousands  of  staff-years  of  experience  as  presented  in  "SPRDE  Functional  Career  Field: 
Critical  Acquisition  Workforce  Data  FY  2013-Q3."(DoD  2013) 

Organizations  have  responded  to  these  challenges  in  a  variety  of  ways,  such  as  offering  extended 
training  and  education  to  their  current  workforce  or  systematically  seeking  to  select  specialty  engineers 
with  promise  as  systems  engineers  and  incorporating  them  into  the  ranks  of  (generalist)  systems 
engineers.  It  is  unknown  if  these  actions  are  producing  the  expected  results  because  there  is  no 
common  understanding  of  the  diverse  roles  that  systems  engineers  play;  how  they  are  selected  and 
evaluated;  what  competencies  are  most  important  for  different  roles;  how  to  evaluate  effectiveness; 
how  experience  impacts  effectiveness.  These  and  many  other  insights  will  be  critical  to  maintaining  and 
growing  the  systems  engineering  workforce  in  the  US  DoD  and  DIB. 

In  order  to  provide  these  insights,  the  Systems  Engineering  Research  Center  (SERC),  a  US  DoD  University 
Affiliated  Research  Center  (UARC),  has  initiated  the  Helix  Project  to  investigate  the  "DNA"  of  the  defense 
systems  engineering  workforce.  The  US  Deputy  Assistant  Secretary  of  Defense  for  Systems  Engineering 
(DASD(SE))  and  the  Systems  Engineering  Division  of  the  National  Defense  Industrial  Association  (NDIA- 
SED)  jointly  sponsor  Helix.  To  ensure  Helix  delivers  the  greatest  value  and  to  help  Helix  obtain  access  to 
the  necessary  data,  the  DoD  and  NDIA-SED  have  formed  the  Helix  Advisory  Panel  (HAP)  with 
representatives  from  both  organizations. 

Helix  is  a  multi-year  longitudinal  research  project,  which  proposes  to  gather  data  from  many  DoD  and 
DIB  organizations  through  a  combination  of  techniques,  including  interviews  with  hundreds  of  systems 
engineers.  Helix  may  reach  beyond  DoD  and  the  DIB  to  gather  data  from  other  types  of  organizations  as 
well. 

This  technical  report  is  the  first  published  by  Helix,  presenting  early  findings  based  on  interviews 
conducted  with  over  100  systems  engineers  from  seven  organizations  in  2013.  The  remainder  of  the 
report  is  organized  as  follows: 

•  Section  2  defines  the  research  questions  that  Helix  is  trying  to  answer 

•  Section  3  explains  the  research  methodology  deployed 

•  Section  4  identifies  the  various  data  sources  and  explains  how  the  data  is  collected  and  handled 

•  Section  5  discussed  the  methods  used  for  analyzing  the  data 

•  Section  6  elaborates  on  the  interviewee  demographics  and  the  early  findings  obtained  from 
analyzing  data  collected  in  2013 

•  Section  7  summarizes  the  findings  and  how  they  relate  to  the  research  questions 

•  Section  8  presents  planned  future  work. 

This  technical  report  is  the  first  of  several  that  will  detail  an  increasingly  rich  set  of  findings  based  on  an 
ever-growing  set  of  data. 
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2.  Research  Questions 


In  order  to  deepen  the  understanding  of  the  systems  engineering  workforce,  Helix  is  focusing  on  three 
main  research  questions: 

1.  What  are  the  characteristics  of  systems  engineers? 

There  is  no  uniform  way  to  identify  people  who  are  systems  engineers.  Some  have  the  formal 
title  "systems  engineer,"  recognized  as  such  by  an  organization's  personnel  system.  Some 
individuals  perform  traditional  systems  engineering  activities,  but  have  another  title,  such  as 
"systems  analyst"  or  "project  engineer".  Some  perform  a  mix  of  systems  engineering  and 
other  activities.  The  Helix  Project  is  seeking  to  understand  the  current  ways  DoD  and  DIB 
organizations  (and  perhaps  other  organizations)  use  to  identify  individuals  who  perform 
systems  engineering  activities.  For  example,  if  an  individual  spends  20%  of  his  or  her  time 
performing  systems  engineering  activities,  is  that  person  a  systems  engineer?  What  if  he  or 
she  spends  51%  on  systems  engineering  activities?  How  do  the  characteristics  of  systems 
engineers  vary  with  the  mission  of  the  organization,  e.g.  in  acquisition  organizations  versus 
development  organizations?  The  Helix  team  hopes  to  address  these  and  other  nuances  as  the 
research  continues. 

2.  How  effective  are  systems  engineers  and  why? 

One  clear  way  to  address  the  challenges  that  the  systems  engineering  workforce  faces  is  to 
make  existing  and  future  systems  engineers  more  effective.  This  naturally  leads  to  questioning 
how  to  measure  effectiveness  and  which  forces,  such  as  competencies,  attrition,  education, 
culture,  environment,  expectations,  and  experiences,  have  the  greatest  impact  on  the 
effectiveness  of  systems  engineers.  Helix  is  trying  to  answer  those  questions. 

Helix  has  posited  generic  definitions  of  individual  and  workforce  effectiveness,  which  will 
evolve  as  it  collects  organizational  perspectives: 

o  An  individual  systems  engineer  is  effective  when  the  outcomes  for  which  he  or  she 
is  individually  responsible  are  achieved  as  a  result  of  the  systems  engineering 
activities  he  or  she  performs. 

o  An  organization's  systems  engineering  workforce  is  effective  when  the  outcomes  for 
which  it  is  collectively  responsible  are  achieved  as  a  result  of  the  systems 
engineering  activities  it  performs. 

3.  What  are  employers  doing  to  improve  the  effectiveness  of  their  systems  engineers? 

Helix  is  trying  to  understand  how  organizations  manage  their  systems  engineering  workforce 
as  a  whole.  For  example,  are  there  organization-level  policies  governing  the  career 
advancement  of  systems  engineers  separate  from  policies  for  other  engineers?  Are  there 
organizational  initiatives  to  help  young  systems  engineers  get  needed  education,  training, 
mentoring,  and  experience  through  rotational  assignments?  How  well  are  those  initiatives 
working?  Are  there  competency  models  for  systems  engineers  that  guide  personnel  selection 
and  job  assignments?  Helix  is  attempting  to  probe  these  areas  across  dozens  of  US  defense- 
related  organizations  in  both  government  and  industry,  looking  for  patterns  of  best  practice 
and  lesser  practice. 
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3.  Research  Methodology 


3.1  Mixed  Methods  and  Grounded  Theory 

The  nature  of  data  to  be  collected  and  the  analysis  to  be  performed  for  Helix  require  both  quantitative 
and  qualitative  research  methods;  this  is  typically  known  as  mixed  methods  research.  (Creswell  2011) 

Qualitative  research  aims  to  create  or  discover  what  things  are  made  of,  and  what  is  created  or 
discovered  are  called  constructs.  Qualitative  research  is  useful  for  obtaining  insight  into  situations  and 
problems  on  which  one  has  little  knowledge  a  priori.  This  method  is  commonly  used  for  providing  in- 
depth  descriptions  of  procedures,  beliefs  and  knowledge,  including  the  opinions  of  respondents  about 
particular  issues.  Detailed  data  is  gathered  through  open-ended  questions  that,  among  other  things, 
provide  direct  quotations.  The  interviewer  is  an  integral  part  of  the  investigation. 

Qn  the  other  hand,  quantitative  research  begins  once  the  constructs  are  in  hand.  It  attempts  to  gather 
data  by  objective  methods  to  provide  information  about  relations,  comparisons,  and  predictions  and 
attempts  to  remove  the  investigator  from  the  investigation.  In  the  context  of  the  Helix  project,  some 
quantitative  research  can  be  performed  based  on  initial  data  collection;  e.g.,  demographic  information 
on  systems  engineers  collected  from  resumes  or  from  organizational  data  sources.  Some  quantitative 
research  can  only  be  performed  after  constructs  are  created  utilizing  qualitative  research;  e.g., 
understanding  the  strength  of  important  characteristics  and  competencies  of  effective  systems 
engineers  within  a  certain  population. 

Since  the  Helix  project  did  not  presuppose  a  specific  theory  of  systems  engineers  (important 
characteristics,  competencies,  relationships,  etc.),  a  grounded  theory  approach  was  adopted.  Grounded 
theory  was  developed  in  the  social  sciences  as  a  method  for  developing  theory  that  is  grounded  in  data 
that  is  systematically  gathered  and  analyzed.  (Goulding  2002)  Rather  than  beginning  with  a  hypothesis, 
the  first  step  is  data  collection.  From  the  collected  data,  key  points  are  marked  with  a  series  of  codes, 
which  are  extracted  from  the  text  and  numeric  information.  The  codes  are  grouped  into  similar 
concepts.  From  these  concepts  constructs  (categories)  are  formed,  which  are  the  building  blocks  of  a 
theory.  This  approach  is  unusual  in  engineering  research,  where  a  researcher  traditionally  begins  with  a 
theoretical  framework  that  he  or  she  applies  to  the  phenomenon  to  be  studied.  While  each  of  the  Helix 
team  members  has  knowledge  and  experience  in  systems  engineering,  use  of  grounded  theory  for  Helix 
is  important  to  the  project  because  it  helps  to  remove  the  team's  biases  and  allows  for  more  open 
investigation  and  the  discovery  of  "surprising"  information.  The  team  was  concerned  that  an  overly- 
structured  approach  would  have  a  higher  potential  for  missed  insights. 

3.2  Overall  Research  Process 

An  overall  process  was  defined  for  conducting  Helix  research  following  the  grounded  theory  approach, 
as  illustrated  in  Figure  1  below. 

The  process  is  intended  to  be  iterative.  The  frequency  and  coverage  of  the  individual  loops  is  still 
evolving.  Steps  A  and  B  are  repeated  for  every  organization.  Step  C  is  performed  after  collecting  data 
from  several  organizations  -  currently,  part  of  the  data  collected  has  been  analyzed  to  produce 
preliminary  findings.  Additional  analysis  is  being  conducted  to  improve  previous  findings  and  to  identify 
new  ones.  Steps  D  and  E  produce  periodic  public  and  private  reports.  This  is  the  first  public  report. 
Private  organizational  reports  will  be  prepared  once  there  are  sufficient  participating  organizations  to 
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protect  the  identity  of  individual  organizations.  Step  F  is  being  conducted  incrementally  every  time  Steps 
B  and  C  are  executed. 


F2.  Identify  Updates  to 
Institutional  Data  Requests 


F3.  Identify  Updates  for 
Data  Collection  Approach 


Figure  1.  Flelix  Research  Process 
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4.  Data  Sources,  Collection,  and  Handling 


Helix  has  been  collecting  two  primary  forms  of  data  from  each  organization:  written  institutional  data, 
such  as  organizational  policies  about  systems  engineers,  and  interview  data  from  face-to-face  meetings 
with  systems  engineers  and  those  who  work  with  systems  engineers. 


4.1  Organizational  Data 

Organizational  data  has  been  obtained  from  various  artifacts  that  each  participating  organization  shares 
with  Helix,  typically  under  a  non-disclosure  agreement  (NDA).  Organizational  representatives  send  the 
Helix  team  internal  documents  relevant  to  the  performance  of  systems  engineering  activities  and  the 
management,  development,  retention,  etc.,  of  systems  engineers.  The  Helix  team  then  analyzes  the 
artifacts,  coordinates  with  organizational  representatives  to  clarify  any  issues  regarding  the  contents  of 
those  documents,  and  produces  an  organizational  summary  that  is  studied  in  advance  of  the  face-to- 
face  interviews  and  informs  some  of  the  interview  questions.  The  Helix  team  requests  the  following 
types  of  data  from  each  organization: 

1.  The  charter  or  primary  purpose  of  the  organization; 

2.  The  primary  business  of  the  organization,  including  revenue,  primary  customer,  organization 
chart,  and  types  of  products  and  services  delivered; 

3.  The  total  number  of  employees  in  the  organization  in  each  year  since  2009,  divided  into 
engineers  and  non-engineers,  including  the  number  of  people  hired  and  departed; 

4.  The  total  number  of  systems  engineers  in  the  organization  in  each  year  since  2009  including  the 
number  of  people  hired  and  departed,  along  with  an  explanation  of  how  the  organization 
defines  "systems  engineer"; 

5.  A  characterization  of  the  population  of  systems  engineers  with  respect  to  highest  college 
degree,  number  of  years  of  professional  experience,  number  of  years  of  experience  as  a  systems 
engineer,  age,  gender,  title  or  rank  (such  as  systems  engineer,  senior  systems  engineer,  chief 
systems  engineer,  etc.  using  whatever  titles  or  ranks  that  are  part  of  the  human  resources 
system),  and  years  to  retirement  eligibility; 

6.  The  way  in  which  the  systems  engineers  are  primarily  organized;  e.g.,  in  a  matrix  structure  or 
project  structure; 

7.  Major  previous  or  current  organizational  initiatives  to  improve  the  quality  or  quantity  of  systems 
engineers;  and 

8.  Policies  that  are  particularly  relevant  to  systems  engineers,  including  organizational  competency 
model  and  career  paths. 

Over  time,  the  requested  information  will  undoubtedly  evolve.  No  organization  is  expected  to  provide 
all  the  requested  data.  The  data  may  not  be  readily  available  or  the  organization  may  not  wish  to  share 
it  with  the  Helix  team.  Nevertheless,  with  enough  institutional  data  from  a  sufficient  number  of 
organizations.  Helix  will  be  able  to  address  its  research  questions. 

For  very  large  organizations,  the  Helix  team  may  interview  several  distinct  segments  of  the  organization. 
For  example,  for  the  US  DoD,  the  DoD-wide  approach  may  be  the  same,  but  there  may  be  different 
policies  or  implementations  within  each  of  the  Services.  For  large  DIB  organizations,  it  is  possible  that 
different  branches  may  have  different  rules  governing  the  SE  workforce,  hire  people  with  different 
backgrounds,  offer  different  forms  of  training,  etc.  Understanding  the  diversity  within  a  single 
organization  should  provide  further  insight  into  the  complexities  and  subtleties  of  the  diverse  roles 
systems  engineers  play  throughout  the  community. 
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It  must  be  noted  that  the  Helix  team  has  had  unexpected  difficulty  in  collecting  this  data  from 
participating  organizations.  The  major  issue  -  which  is  further  discussed  in  Sections  5  and  6  -  seems  to 
be  that  most  organizations  do  not  have  a  clear  method  for  identifying  or  even  defining  systems 
engineers.  This  makes  it  very  difficult  to  provide  demographics  on  systems  engineering  population 
within  an  organization  and  even  complicates  the  identification  of  relevant  policies.  The  Helix  team  will 
continue  to  work  with  organizational  points  of  contact  (POCs)  to  obtain  this  information.  However,  it  is 
currently  and  may  possibly  remain  a  gap  in  the  information  collected. 


4,1,1  Vision  for  Participation 

To  address  these  research  questions,  Helix  has  begun  collecting  data  from  organizations  in  both  DoD 
and  the  DIB  and,  over  time,  hopes  to  collect  data  from  dozens  of  additional  organizations  and  perhaps  a 
thousand  individual  systems  engineers  from  those  organizations.  As  of  the  writing  of  this  report.  Helix 
has  collected  data  from  seven  organizations.  Others  have  agreed  to  participate  in  the  coming  months, 
and  several  more  are  considering  participation.  In  2014  or  2015,  the  scope  of  Helix  could  be  expanded 
beyond  US  DoD  and  DIB  to  include  organizations  from  other  business  segments  in  the  US  or 
organizations  from  outside  the  US;  however,  such  expansion  has  not  yet  been  determined. 

It  is  the  vision  of  Helix  to  gather  sufficient  data  that  would  truly  represent  the  global  systems 
engineering  community,  and  the  many  variations  within  it  in  terms  of: 

•  The  nature  of  systems  (engineering  and  otherwise), 

•  Geographical  location, 

•  Government  versus  commercial  organizations, 

•  The  scale  of  systems  or  systems-of-systems,  and 

•  Domains  such  as  defense,  finance,  transportation,  communications,  and  healthcare. 

Current  data  focuses  only  on  defense  systems,  but  even  within  the  defense  application,  participating 
organizations  to  date  have  worked  across  several  domains  and  several  have  commercial  business 
sectors.  In  2014,  the  Helix  team  will  attempt  to  incorporate  a  greater  breadth  of  domains. 

In  addition  to  growing  the  diversity  of  organizations  with  respect  to  the  above  criteria,  the  Helix  team 
wants  to  increase  the  representation  within  those  criteria;  e.g.,  collecting  data  from  all  of  the  largest  DIB 
organizations  who  build  and  integrate  major  platforms  (often  referred  to  as  the  Tier  1  companies),  many 
of  their  largest  suppliers  (the  Tier  2  companies),  and  even  some  of  the  component  suppliers  (the  Tier  3 
companies).  This  is  in  addition  to  the  many  different  organizations  within  the  DoD  itself;  e.g.,  collecting 
data  from  all  the  Services  and  from  organizations  that  work  in  different  parts  of  the  lifecycle,  such  as 
laboratories  that  work  at  the  front-end  of  the  lifecycle  and  depots  that  work  at  the  tail  end  of  the 
lifecycle.  By  building  a  sample  that  includes  multiple  examples  for  each  of  these  criteria,  the  Helix  team 
believes  it  will  eventually  be  able  to  produce  generalizable  results. 


4,1,2  Value  to  Participating  Organizations 

Helix  research  findings  will  be  published  in  reports,  papers,  and  presentations,  providing  important 
insights  to  the  broad  systems  engineering  community.  However,  the  most  immediate  beneficiaries  will 
be  the  individual  participating  organizations,  which  will: 

1.  Deepen  their  understanding  of  their  systems  engineering  workforce  over  time, 

2.  Identify  gaps  and  where  to  focus  investments  to  improve  the  systems  engineering  workforce, 

3.  Benchmark  their  systems  engineering  workforce, 

4.  Deepen  their  understanding  of  what  makes  the  workforce  effective,  and 
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5.  Receive  recommendations  on  how  to  improve  the  effectiveness  of  their  systems  engineers. 

Helix  has  received  feedback  that  just  by  participating  in  the  project,  some  participating  organizations 
have  begun  to  reassess  their  approaches  to  managing  and  growing  their  systems  engineering  workforce. 

4.2  Interview  Data 

Interviews  are  conducted  with  a  diverse  cross  section  of  individuals  in  each  organization.  Helix  seeks 
people  who: 

•  Are  knowledgeable  about  the  characteristics  of  systems  engineers  across  the  organization, 

•  Perform  systems  engineering  on  programs  including  both  senior  and  junior  personnel, 

•  Supervise  systems  engineers,  and 

•  Use  the  products  of  systems  engineers. 

Interview  questions  are  related  to  the  three  main  research  questions,  and  are  only  meant  to  initiate  and 
guide  discussions  on  various  topics  of  interest  to  Helix.  The  interviews  are  conversations  facilitated  by 
the  interviewer  rather  than  a  means  to  collect  responses  to  a  pre-determined  set  of  questions.  Examples 
of  the  Helix  interview  questions  can  be  found  in  Appendix  A. 


4.3  Protocols  and  Approvals 

Due  to  the  nature  of  the  data  to  be  handled  in  Helix,  various  approvals  had  to  be  obtained  and 
agreements  signed  to  ensure  proper  handling  of  data  and  protection  of  identities  of  individuals  and 
organizations: 

1.  Approval  from  Institutional  Review  Board  (IRB).  Stevens  Institute  of  Technology  has  an  IRB  to 
review  all  research  that  involves  human  subjects.  All  members  of  the  research  team  had  to 
complete  training  on  Social  &  Behavioral  Research  that  included  ethical  principles,  human 
subjects,  privacy,  confidentiality,  and  other  topics.  The  IRB  reviewed  the  research  methodology, 
data  collection  instruments,  data  handling  protocols,  and  all  relevant  information  before 
approving  the  Helix  team  to  conduct  research. 

2.  Approval  from  DoD.  The  project  sponsors  also  required  a  review  of  the  research  methodology 
by  a  DoD  agency  to  confirm  that  Helix  satisfies  all  applicable  DoD  requirements. 

3.  NDAs  with  participating  organizations.  These  are  executed  with  participating  organizations  as 
per  mutually  agreed  terms.  Data  is  then  shared  with  the  research  team;  participants  are  assured 
of  privacy  and  confidentiality  of  their  identities  and  the  information  they  provide. 

4.  Consent  Form  from  participating  individuals.  All  those  who  are  interviewed  sign  a  consent  form 
that  explains  their  rights  and  confirms  that  their  participation  is  voluntary. 


4.4  Data  Security 

The  nature  and  quantity  of  the  data  to  be  collected  warranted  sophisticated  measures  for  data  security 
to  be  established  before  any  data  could  be  collected.  It  was  expected  that  almost  all  data  would  be 
handled  in  electronic  format,  though  they  may  be  printed  as  hard  copies  for  use  by  researchers  or  for 
archival  purposes.  For  physical  data,  it  was  mandated  that  all  data  be  under  lock-and-key  at  all  times 
except  when  being  used  actively,  in  which  case  it  is  under  the  direct  supervision  and  control  of  Helix 
personnel.  For  electronic  data,  appropriate  security  solutions  were  identified  for  the  following  scenarios: 
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1.  Data  exchange  over  email.  Data  would  typically  be  received  from  individuals  and  organizations 
via  email;  and  the  geographically  dispersed  nature  of  the  research  team  meant  that  data  would 
be  frequently  exchanged  over  email.  Files  sent  between  members  of  the  Helix  team  were 
encrypted  when  transmitted  using  a  commercial  software  tool. 

2.  Data  storage  on  researcher's  machines.  To  ensure  secure  handling  of  active  data  on  the 
researcher's  computer,  a  software  package  called  TrueCrypt  was  chosen.  TrueCrypt  creates  a 
secure  volume  in  the  local  hard  drive,  which  may  then  be  mounted  when  needed.  The  entire  file 
system  within  the  volume  is  encrypted,  and  on-the-fly  encryption  ensures  automatic  encryption 
just  before  saving  a  file  and  automatic  decryption  just  after  it  is  loaded  without  any  user 
intervention. 

3.  Data  storage  on  Stevens'  servers.  Sakai  Gateway  is  a  secure  private  space  provided  by  Stevens 
Institute  of  Technology  to  its  researchers.  All  sensitive  files  stored  on  Sakai  are  encrypted. 
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5.  Data  Analysis 


5.1  Assumptions 

Several  fundamental  assumptions  underlie  the  Helix  study  and  these  influence  the  research  methods, 
data  analysis,  and  interpretation  of  the  findings.  Though  some  may  seem  obvious,  it  is  important  to 
state  these  assumptions,  the  risks  if  these  assumptions  prove  false,  and  how  the  Helix  team  is  mitigating 
these  risks.  Current  assumptions  include: 

•  Validity  of  Responses.  There  is  an  assumption  that  interviewees  provide  valid  responses;  that  is, 
that  the  individual  making  the  statements  believes  them  to  be  factually  correct  or  is  sharing  his 
or  her  true  views  on  a  subject.  (Anderson  2010) 

The  Helix  team  works  to  ensure  valid  responses  in  several  ways.  First,  the  team  interfaces  with  a 
single  point  of  contact  (POC)  in  each  organization;  each  POC  is  briefed  on  Helix,  including  the 
need  for  open  and  honest  responses  and  the  fact  that  any  organizational  influence  on  the 
participants  (e.g.  "if  they  ask  you  about  this,  here  is  what  the  organization  believes  .  .  .  ")  will 
damage  the  validity  of  the  study.  To  date,  all  POCs  have  understood  and  agreed  with  this. 

All  participants  attend  an  informational  session  in  which  the  Helix  team  discusses  the  need  for 
open  honest  answers.  The  team  confirms  in  the  information  sessions  and  again  before  each 
interview  that  neither  the  Helix  team  nor  the  organization  has  provided  any  incentives  for 
participating  or  given  instructions  on  how  to  respond.  All  participants  are  also  assured 
throughout  the  process  that  their  identities  will  not  be  revealed  and  that  the  team  will  only 
report  anonymous  results.  The  belief  is  that  assurance  of  anonymity  will  help  participants  feel 
more  comfortable  answering  honestly. 

The  team  believes  that  these  measures  are  appropriate  and  sufficient,  but  recognizes  that  it  is 
impossible  to  know  with  certainty  whether  participants  are  being  completely  honest.  However, 
the  team  has  seen  a  diversity  of  information  and  opinions  within  organizations,  which  indicates 
that  it  is  unlikely  that  organizations  are  influencing  the  process.  Finally,  the  team  has  an  internal 
discussion  after  each  interview  as  to  the  apparent  openness  of  the  participants  based  on 
observing  body  language,  eye  contact,  etc.  To  date,  the  team  does  not  have  any  reason  to  doubt 
the  participants  have  answered  in  a  way  that  reflects  their  personal  views. 

•  Participant  Credibility.  There  is  an  assumption  that  individuals  who  participate  in  Helix  are 
effective  and  high  performing  and,  therefore,  may  provide  better  insights  into  the  systems 
engineering  workforce  than  weaker  systems  engineers.  The  team  specifically  requests  that  POCs 
identify  some  of  the  best  systems  engineers  in  the  organization  for  potential  participation  in  the 
study.  If  this  assumption  is  incorrect,  it  may  mean  that  some  depth  or  insights  will  be  missed. 

•  Validity  of  Organizational  Representatives.  Each  POC  is  asked  to  identify  some  individuals  who 
can  speak  for  the  organization  as  a  whole.  The  belief  is  that  these  individuals  really  do  provide 
insight  into  the  organization,  as  they  understand  the  policies,  processes,  and  tools  and  are  also 
familiar  with  many  of  the  larger-level  workforce  issues.  However,  most  POC's  have  not  clearly 
identified  their  "organizational"  interviewees.  Therefore,  the  Helix  team  looks  to  an 
organization's  most  senior  systems  engineers  (herein  called  chief  systems  engineers,  whatever 
their  actual  organizational  title)  to  have  a  broad  knowledge  of  the  way  systems  engineering  is 
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performed  within  the  organization  and  a  pulse  on  the  SE  workforce  as  a  whole  and  considers 
them  to  be  good  systems  engineers.  The  team  considers  these  chief  systems  engineers  to  be 
organizational  representatives  and  highly  values  their  responses. 

The  team  further  assumes  that  these  organizational  representatives  will  not  simply  provide  their 
personal  opinions,  but  will  provide  a  higher-order  voice  for  the  organization.  Their  responses  are 
treated  in  this  manner  -  as  providing  more  insight  at  the  organization  level  and  providing  valid 
perspectives  on  the  population  as  a  whole.  If  this  assumption  is  not  correct,  the  risk  is  that  the 
population-level  findings  will  be  unduly  influenced  by  the  responses  of  these  individuals.  The 
team  believes  that  the  perspectives  of  other  individuals  who  are  asked  to  speak  for  themselves 
helps  to  balance  this  and  to  mitigate  the  risk.  For  example,  the  team  has  observed  on  several 
occasions  that  organizational  representatives  were  well  aware  of  improvement  initiatives  and 
state  that  the  workforce  is  generally  aware  of  these  initiatives;  the  individual  respondents, 
however,  state  that  they  are  less  aware  of  these.  The  individual  contributors  then  balance  these 
types  of  findings.  This  also  becomes  a  point  of  interest  for  the  individual  organizations. 

•  Objectivity  of  Helix  Research  Team.  The  Helix  team  is  taking  every  effort  to  remain  objective, 
neutral,  and  unbiased  in  the  entire  research  process.  The  interview  questions  are  framed  and 
posed  to  participants  in  a  manner  that  should  not  imply  any  prejudice  or  would  lead  them  in  any 
particular  direction.  During  the  interview,  participants  are  encouraged  to  share  their  opinions 
without  being  judged  on  what  they  are  saying  to  be  right  or  wrong.  To  date,  there  have  been  no 
concerns  raised  by  the  participants  or  their  organizations  on  the  manner  in  which  the  interviews 
have  been  collected  or  how  the  participants  have  been  treated. 

The  Helix  team  believes  these  mitigation  approaches  are  currently  adequate  and  will  remain  vigilant  to 

revise  them  as  circumstances  warrant. 


5.2  Analysis  Methods 

Data  is  being  analyzed  using  both  qualitative  and  quantitative  research  methods.  Quantitative  analysis  is 
possible  for  some  of  the  data,  such  as  demographic  information,  including  the  distribution  of  ages  and 
number  of  systems  engineers  in  an  organization,  which  will  be  aggregated  to  different  community 
segments,  such  as  large  versus  small  defense  contractors. 

Some  of  the  more  interesting  analyses  will  be  qualitative  and  will  center  around  the  most  important 
activities  systems  engineers  perform,  what  the  most  important  characteristics  of  systems  engineers  are, 
and  what  makes  systems  engineers  more  or  less  effective. 

A  modified  grounded  theory  approach  is  being  followed  at  this  stage  of  the  research  for  qualitative 
analysis.  A  pre-determined  set  of  interview  questions  is  being  used  by  the  interviewer  as  a  guide,  but 
the  interviews  follow  interesting  lines  of  discussion  triggered  by  comments  from  the  interviewees.  As  a 
result,  the  interviews  conducted  so  far  have  been  based  on  some  expectation  on  responses  (hence  not 
fully  based  on  grounded  theory),  and  the  interviews  explore  the  actual  responses  more  deeply  (hence 
not  fully  structured).  The  direction  of  the  interviews  are  also  informed  by  the  resumes  of  those  being 
interviewed  as  well  as  the  institutional  data  collected  from  their  organizations. 

Data  analysis  begins  with  the  creation  of  an  interview  summary.  If  an  organization  permits  audio 
recording,  transcripts  are  created  that  are  the  basis  for  summaries.  Currently,  the  research  team  is  using 
a  qualitative  analysis  software  package  called  Dedoose.  The  interview  summaries  are  uploaded  to 
Dedoose  and  are  coded.  There  are  multiple  levels  of  coding: 
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•  Direct  responses  to  questions  posed  to  the  interviewees; 

•  Information  that  was  not  in  direct  response  to  any  interview  question; 

•  Patterns  across  interviewees  within  a  particular  organization;  and 

•  Patterns  across  organizations. 

In  addition  to  coding  of  free  text,  characteristics  of  individual  interviewees  and  organizations  are  also 
tabulated  and  cross-linked  with  the  codes  to  provide  deeper  insights. 

The  identity  of  all  interviewees  is  kept  confidential  and  reports  only  include  anonymous,  aggregated 
data.  The  names  of  participating  organizations  are  only  revealed  with  their  permission  and  in  a  manner 
that  will  not  correlate  them  with  research  findings. 


5.3  Analysis  Challenges 

Analysis  to  date  has  been  based  upon  the  interviews  because  institutional  data  has  been  surprisingly 
difficult  to  collect.  The  team  initially  believed  that  institutional  data  would  be  the  primary  component  of 
the  first  report  and  would  help  set  the  context  for  preliminary  findings  from  interview  data.  Most 
organizations  do  not  have  a  clear  method  for  identifying  or  even  defining  systems  engineers.  This  makes 
it  very  difficult  to  provide  demographics  on  the  systems  engineering  population  within  an  organization 
and  even  complicates  the  identification  of  relevant  policies. 

The  Helix  team  had  planned  to  provide  analysis  of  interview  information  in  the  context  of  different  types 
of  organizational  settings.  Because  this  information  is  lacking  in  the  current  analysis,  the  findings  can  not 
be  contextualized  in  this  manner.  The  team  will  continue  to  work  with  POCs  to  build  an  understanding 
of  the  organizaitonal  setting  and  fold  this  into  the  analyses  going  forward. 

This  is  further  complicated  by  the  fact  that  only  seven  organizaitons  have  been  visited  to  date.  The  small 
number  makes  it  impossible  to  provide  organizational  classification  while  protecting  the  privacy  of 
participating  organizations.  As  the  pool  of  participating  organizations  grows,  the  team  will  be  able  to 
provide  more  details  by  organization  size,  domain,  technology  focus,  etc. 
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6.  Early  Findings 


Enough  data  has  been  collected  to  report  early  findings,  which  fall  into  two  categories:  1)  interviewee 
demographics  and  2)  findings  from  interviews.  The  interviewee  demographics  describe  information 
about  the  sample  of  individuals  who  have  participated  in  Helix.  The  findings  from  interview  data  include 
characteristics  and  competencies  of  effective  systems  engineers;  career  paths  for  senior  systems 
engineers;  greatest  contributions  of  systems  engineers;  factors  that  positively  and  negatively  impact  the 
effectiveness  of  sytems  engineers;  and  potential  risks  to  the  systems  engineering  workforce. 

These  findings  are  based  on  the  data  that  has  been  collected.  Therefore,  they  may  not  be  generalizable 
to  the  entire  systems  engineering  population  within  the  DoD  and  DIB.  As  Helix  continues  to  visit  more 
organizations  and  conduct  more  interviews,  it  is  expected  that  a  more  representative  sample  will  be 
developed  such  that  findings  may  become  generalizable.  Further,  in  this  report,  no  organizational-level 
findings  are  being  reported  since  only  a  limited  number  of  organizations  in  DoD  and  DIB  have  been 
visited  to  date  and  due  to  the  difficulties  in  gaining  access  to  organizational  data,  as  discussed  in 
Sections  4  and  5  above. 

This  report  presents  findings  in  an  anonymous  aggregated  manner  so  that  no  observation  may  be  traced 
back  to  an  individual  or  to  an  organization.  In  light  of  this,  there  is  data  from  the  interviews  that  has  not 
yet  been  analyzed  and  not  included  in  this  report.  Future  reports  from  Helix  will  update  the  current 
findings  as  well  as  offer  new  findings  at  the  individual  and  organizational  levels. 


6.1  Interviewee  Demographics 

This  section  discusses  the  demographics  of  the  sample  of  individuals  interviewed  to  date  for  the  Helix 
project.  It  represents  110  systems  engineers  from  seven  organizations.  There  were  two  additional 
interviewees  who  worked  with  systems  engineers;  these  interviews  are  included  in  interview  findings, 
but  not  in  the  demographic  information  below. 

6.1.1  Sample  Population 

During  2013,  seven  organizations  were  visited  and  112  participants  were  interviewed  as  part  of  Helix. 
Figures  2  and  3  indicate  the  number  of  interview  sessions  conducted  per  organization,  and  the  total 
number  of  interviewees  in  each  organization. 


Interview  Sessions  per  Organization 


Interviewees  per  Organization 


Org.  1  Org.  2  Org.  3  Org.  4  Org.  5  Org.  6  Org.  7 


Org.  1  Org.  2  Org.  3  Org.  4  Org.  5  Org.  6  Org.  7 


Figure  2.  Interview  Session  per  Organization  Figure  3.  Interviewees  per  Organization 

There  were  one  to  four  interviewees  in  each  interview  session,  as  illustrated  in  Figure  4.  Initially,  the 
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plan  was  to  conduct  only  individual  interviews  but  due  to  scheduling  constraints,  more  than  one 
interviewee  was  included  in  most  interview  sessions.  During  those  interviews,  it  was  observed  that  the 
dynamic  between  the  participants  enhanced  their  responses.  So,  it  was  very  effective  to  schedule  two  or 
three  interviewees  per  session.  In  some  cases,  four  interviewees  were  scheduled  in  a  session  due  to 
constraints,  but  that  was  not  preferred  because  it  made  it  difficult  to  give  enough  time  to  gather  data 
from  every  participant  and  to  discuss  several  topics. 


Interviewees  per  Session 


Size  of  Interview  Team  per  Session 


4  interviewees 

4% 


3  interviewees 
36% 


4  interviewers 
2% 


3  Interviewers 
55% 


2  Interviewers 
43% 


Figure  4.  Interviewees  per  Session 


Figure  5.  Size  of  Interview  Team  per  Session 


Figure  5  illustrates  the  size  of  the  interview  team  for  the  interview  sessions  conducted  so  far.  Initially, 
the  size  of  the  interview  team  was  three,  to  include  one  lead  interviewer  and  two  other  researchers  to 
take  notes  and  ask  additional  questions.  As  the  interview  process  matured  over  time,  it  was  most 
effective  to  limit  the  size  of  the  interview  team  to  two,  and  this  is  the  preferred  size  of  the  interview 
team  going  forward.  A  few  interview  sessions  had  four  members  on  the  interview  team,  only  for  the 
benefit  of  new  incoming  researchers  to  observe  the  interview  process. 

Since  the  interviews  were  conducted  as  conversations,  and  the  data  being  collected  from  the  interviews 
were  highly  unstructured  and  massive  in  size,  it  was  highly  preferred  to  record  audio  from  the 
interviews  and  to  use  the  transcripts  from  those  interviews  for  further  analysis.  But  this  was  first  an 
organizational  decision,  and  then  the  choice  of  the  individual  participant.  As  illustrated  in  Figure  6,  while 
many  interviews  were  recorded,  40%  of  the  interviews  could  not  be  recorded  largely  due  to 
organizational  decisions. 


Audio  Recording 


Figure  6.  Audio  Recording 


interview  Mode 


Telecon 

4% 


Figure  7.  Interview  Mode 


The  intent  of  Helix  was  to  conduct  face-to-face  interviews,  in  the  expectation  that  this  would  provide 
the  best  environment  for  the  interviewees  to  participate  in  the  semi-structured  interview  sessions  and 
for  the  interview  team  to  respond  to  physical  cues  and  to  direct  the  interview  in  a  comfortable  engaging 
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manner.  As  illustrated  in  Figure  7,  a  few  interviews  were  conducted  over  the  telephone  due  to  the 
locations  of  the  interviewees  and  the  number  of  interview  sessions  scheduled  at  that  location.  As  best 
as  the  interview  team  could  sense,  most  interviewees  did  exhibit  confidence  in  the  interview  team  and 
provided  honest,  sincere,  and  elaborate  responses.  However,  there  were  some  interviewees  who  by 
nature  spoke  less  and  it  was  difficult  to  extract  much  useful  data  from  them.  Helix  will  continue  to  use 
face-to-face  interviews  as  the  primary  mode  of  data  collection  from  individuals. 

As  noted  earlier.  Helix  interviewed  112  participants  but  2  of  them  were  HR  personnel,  and  not  systems 
engineers.  In  the  future.  Helix  will  interview  more  such  non-systems  engineers:  those  who  recruit,  work 
with,  or  manage  systems  engineers  who  themselves  are  not  systems  engineers.  The  demographic 
information  presented  in  the  rest  of  this  section  is  based  on  data  from  the  110  systems  engineers  only. 
An  'unknown'  column  is  included  in  figures  as  appripriate,  to  account  for  interviewees  about  whom  the 
requried  data  is  not  available.  All  findings  reported  in  subsequent  sections  are  also  based  on  data  from 
these  110  interviewees  only. 

The  participants  in  the  Helix  interviews  were  predominantly  male,  as  illustrated  in  Figure  8.  The 
participants  were  always  chosen  by  the  respective  organizations  and  not  by  the  Helix  team  which  stated 
some  expectations  on  the  diversity  of  participants.  Figure  9  illustrates  the  distribution  of  the  participants 
by  age.  It  must  be  noted  that  age  was  not  explicitly  requested  by  Helix  or  provided  by  participants,  but 
an  indicative  age  was  calculated  based  on  the  graduation  year  of  their  bachelor  degrees.  Hence,  the 
exact  ages  may  be  off  by  a  year  or  two,  but  the  ranges  are  expected  to  be  largely  true. 


Distribution  by  Gender 


Distribution  by  Age 


Female 

15% 


Male 

85% 


<25  26-35  36-45  46-55  56-65  >65  unknown 


Figure  8.  Interview  Sample  by  Gender  Figure  9.  Interviewee  Distribution  by  Age 

There  is  insufficient  data  to  consider  these  distributions  on  gender  and  age  to  represent  the  entire 
population  of  systems  engineers  in  the  organizations  or  across  DoD  or  DIB.  This  only  represents  the 
sample  that  participated  in  the  Helix  interviews. 

Among  the  participants,  more  than  50%  of  the  participants  had  a  master's  degree  as  their  highest 
degree,  as  illustrated  in  Figure  10.  But  it  is  also  important  to  note  that  about  30%  of  the  participants  did 
not  pursue  any  formal  education  beyond  a  bachelor's  degree.  About  5%  of  them  had  a  doctorate 
degree. 
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Highest  Degree 


Figure  10.  Highest  Degree  Attained  for  Sample  Population 

The  majors  of  the  bachelors  degrees  indicate  a  spread  among  engineering  degrees,  the  highest  in 
electrical  engineering  and  related  majors  followed  by  mechanical  engineering,  as  illustrated  in  Figure  11. 
This  may  be  a  reflection  on  the  nature  of  systems  being  engineered  by  the  organizations  they  work  for 
and  hence  cannot  be  assumed  to  be  true  across  DoD  or  DIB.  It  must  be  noted  that  some  participants 
have  non-engineering  bachelor's  degrees  as  well. 


Bachelor's  Degrees 


Master's  Degrees 


Figure  11.  Bachelor's  Degrees  Figure  12.  Master's  Degrees 

Figure  12  illustrates  the  majors  for  master's  degrees.  It  must  be  noted  that  not  every  participant  had  a 
master's  degree,  and  some  participants  had  two  master's  degrees.  Most  master's  degrees  were  in 
electrical  engineering  and  related  majors.  Thirteen  participants  possessed  a  masters  degree  in  systems 
engineering.  Among  those  interviewed,  seven  of  them  held  a  PhD  degree,  and  their  thesis 
specializations  were  diverse: 

•  Aeronautical  Engineering 

•  Anthropology 

•  Atmospheric,  Oceanic  and  Space  Sciences 

•  Electrical  Engineering 

•  Geotechnical  Engineering 

•  Mechanical  Engineering 

The  interviewee  demographics  presented  in  this  section  illustrate  the  sample  from  which  data  has  been 
collected  through  interviews.  The  findings  presented  in  the  subsequent  sections  are  representative  of 
this  sample  only,  and  may  not  be  generalized  beyond  this  to  represent  the  larger  population  of  systems 
engineers. 


December,  2013 


Helix  Technical  Report 


23 


6.1.2  A  First  Look  at  Senior  Systems  Engineers 


In  view  of  the  systems  engineering  workforce-related  challenges  to  which  Helix  comes  as  a  response,  it 
was  of  interest  to  find  out  more  about  the  senior  systems  engineers  who  are  close  to  attaining 
retirement  eligibility.  The  challenge  is  that  while  it  was  often  difficult  to  identify  a  "systems  engineer"  in 
an  organization  (see  further  discussion  in  Section  6.2.5  below),  it  was  even  more  difficult  to  identify  a 
"senior  systems  engineer". 

As  a  preliminary  step  to  understanding  the  more  senior  population,  20  systems  engineers  have  been 
identified  from  the  current  interviewee  population  based  on  a  combination  of  their  job  title,  years  of 
experience,  and  nature  of  responsibilities.  For  this  report,  the  Helix  team  has  examined  their  education 
and  high-level  work  experiences. 


Figure  13  illustrates  the  education  timeline  of  the  20  senior  systems  engineers.  For  each  of  them,  the 
type  of  degree  they  obtained  and  the  year  in  which  they  obtained  it  are  indicated. 


Senior  Systems  Engineers:  Education  Profile 
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Figure  13.  Education  Profile  of  20  Senior  Systems  Engineers 


A  number  of  observations  can  be  made  from  Figure  13,  illustrating  the  variety  in  this  data: 

•  Three  of  the  senior  systems  engineers  interviewed  have  PhDs,  but  as  indicated  in  Section  6.1.1, 
none  of  these  degrees  are  in  systems  engineering. 

•  Six  of  them  have  only  a  bachelor's  degree  as  their  formal  education. 

•  Seven  have  obtained  their  master's  degrees  almost  immediately  following  their  bachelor's 
degrees,  while  the  other  individuals  with  master's  degrees  have  gathered  years  of  work 
experience  before  getting  a  master's  degree. 

•  Between  them,  these  20  senior  systems  engineers  have  earned  39  degrees.  The  most  common 
areas  of  study  were  electrical  engineering  (16  degrees^)  and  mechanical  engineering  (8 
degrees);  other  areas  of  study  included  computer  science/software  engineering  and 


’  Note:  Some  of  these  degrees  were  slightly  different  fields,  like  “eiectricai,  eiectronics,  and  communication”. 
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aeronautical  engineering  (both  3  degrees).  The  other  areas  of  study  were  often  specific  to  the 
domain  of  application  or  a  hard  science  (e.g.  physics).  A  few  degrees  were  non-technical  (e.g. 
MBA,  music). 

In  terms  of  experiences  of  these  senior  systems  engineers,  analysis  is  ongoing  due  to  the  wide  variety  of 
experiences  across  this  subsample.  The  Helix  team  is  hoping  to  identify  patterns  with  respect  to  the 
number  of  years  of  "systems  engineering"  experience  vs.  number  of  years  of  "specialty  engineering" 
experience;  the  number  of  years  experience  in  the  current  organization  vs.  number  of  years  spent  in 
other  organizations  including  military  service;  the  breadth  of  exposure  to  different  types  of  systems  and 
different  phases  of  the  systems  engineering  life  cycle;  and  the  sizes  of  their  respective  current  teams  (as 
peers  or  subordinates).  Unfortunately,  no  patterns  have  emerged  to  date.  However,  the  team  believes 
that  as  the  sample  size  grows,  patterns  may  emerge.  The  team  will  also  follow  up  with  interviewees  and 
expects  to  collect  additional  information  about  these  experience  profiles  during  follow  up. 


6.2  Initial  Findings  from  Interviews 

There  were  several  insights  gained  from  the  interviews  conducted  to  date.  For  this  report,  the  team  has 
focused  on  the  patterns  of  responses  to  capture  points  that  are  consistent  across  the  sample.  These 
have  been  grouped  into  the  following  six  categories,  and  discussed  in  more  detail  in  the  following  sub¬ 
sections: 

1.  The  most  important  traits  and  competencies  of  effective  systems  engineers, 

2.  The  greatest  contributions  of  systems  engineers, 

3.  What  makes  systems  engineers  most  effective, 

4.  What  makes  systems  engineers  least  effective,  and 

5.  Potential  risks  to  the  systems  engineering  workforce. 

It  should  be  noted  that  there  are  other  findings  available  from  the  data  already  collected  which  could 
not  be  included  in  this  report  because  of  available  time  or  because  they  are  integrally  related  to  a 
specific  domain  and,  therefore,  would  be  tied  to  specific  organizations.  These  additional  findings  will  be 
included  in  future  reports  along  with  new  findings  that  emerge  as  the  size  of  the  sample  population 
grows  and  as  more  data  is  collected  and  analyzed. 


6.2.1  The  Most  Important  Characteristics  and  Technical  Competencies  of  Effective 
Systems  Engineers 

The  first  research  question  that  Helix  is  exploring  is  to  understand  the  characteristics  of  systems 
engineers.  This  would  be  critical  for  various  workforce  development  initiatives  and  help  organizations 
decide  the  right  mix  of  training/education  methods  to  recruit  and  improve  their  workforce,  to  produce 
effective  systems  engineers.  This  will  also  help  individuals  recognize  and  address  their  strengths  and 
weaknesses  to  become  effective  systems  engineers. 

While  drawing  out  these  characteristics  from  participants  from  interviews,  in  most  cases,  it  was  non¬ 
technical  characteristics  or  "soft  skills"  that  were  provided.  This  led  to  further  discussions  on  the 
importance  of  technical  competencies  for  systems  engineers,  and  the  reasons  they  were  considered 
important  or  not. 

6.2. 1.1  Characteristics 

Interviewees  were  asked  to  identify  and  provide  further  insight  into  the  most  important  characteristics 
of  effective  systems  engineers.  To  help  draw  out  their  responses,  they  were  encouraged  to  think  about 
the  best  systems  engineers  they  knew  or  worked  for.  Based  on  the  responses  received,  five  broad 
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themes  have  been  identified  and  are  elaborated  below.  Individually,  many  of  these  characteristics  may 
be  required  of  any  technical  leader  or  decision  maker.  However,  the  combination  of  these  and  some 
subtle  nuances  in  the  nature  of  these  characteristics  and  why  they  are  needed  in  a  systems  engineer, 
are  insightful.  Also,  these  characteristics  are  not  mutually  exclusive  but  rather  support  and  complement 
each  other.  These  themes  discussed  below  are  expected  to  mature  with  further  data  collection  and 
analysis  and  are  expected  to  support  the  building  a  theory  of  systems  engineers. 

6. 2. 1.1.1  Paradoxical  Mindset 

Looking  at  the  all  the  characteristics  that  the  participants  listed,  it  was  observed  that  many 
characteristics  were  actually  contradictory  to  each  other.  Upon  further  discussion  with  the  participants 
and  deeper  analysis,  it  was  clear  that  effective  systems  engineers  do  possess  seemingly  opposite 
characteristics  simultaneously  and  not  just  as  a  "balance"  between  the  two  extremes.  In  elaborating  this 
mindset,  an  interviewee  said  "systems  engineering  is  not  for  everyone"  and  that  requires  a  "unique 
combination  of  left  brain  -  right  brain  working  together."  Various  such  contradictory  characteristics  are 
identified  below;  in  some  cases  both  were  identified  by  the  same  interviewee  and  by  different 
interviewees  in  some  others. 

•  Big  Picture  Thinking  and  Attention  to  Detail.  Also  referred  to  as  "holistic  thinking"  or  "systems 
thinking",  "big  picture  thinking"  was  commonly  identified  as  a  very  important  and  essential 
characteristic  of  systems  engineers.  An  interviewee  said,  "the  difference  between  a  systems 
engineer  and  a  design  engineer  is  that  a  design  engineer  doesn't  have  to  see  the  big  picture." 
"Big  Picture  Thinking"  relates  to  looking  across  dimensions  of  space  and  time  with  respect  to  the 
system  of  interest  and  its  lifecycle.  At  the  same  time,  a  systems  engineer  is  also  required  to  pay 
attention  to  the  detail.  An  interviewee  elaborated  "there  are  some  people  who  can  see  through 
a  microscope  and  others  who  are  more  visionary.  A  good  systems  engineer  has  some  elements 
of  both". 

•  Strategic  and  Tactical.  Systems  engineers  need  to  be  strategic,  focused  on  the  end  result  of 
"vision"  for  the  system,  but  also  need  to  handle  the  tactical  day-to-day  activities  and  decisions 
required  to  reach  that  vision.  They  must  also  be  able  to  appreciate  "how  what  is  done  today  is 
going  to  affect  things  downstream". 

•  Analytic  and  Synthetic.  A  "big  picture  thinking"  mindset  can  be  associated  with  the  ability  to  be 
synthetic  and  to  be  able  to  bring  together  and  integrate  different  pieces  of  a  puzzle  together. 
But  a  systems  engineer  also  needs  to  be  analytic  and  be  able  to  break  down  a  big  picture  into 
smaller  pieces. 

•  Courageous  and  Humble.  Systems  engineers  need  to  be  courageous  as  a  leader  and  decision 
maker,  but  they  also  need  to  be  humble  are  recognize  the  fact  that  there  others  who  are 
experts  in  individual  areas  of  specialization.  They  are  confident  in  their  abilities  with  a  "lack  of 
pride"  and  "don't  feel  the  need  to  be  defensive".  They  are  also  willing  to  accept  their  mistakes. 

•  Methodical  and  Creative.  Systems  engineers  need  to  be  disciplined,  organized,  diligent,  and 
methodical  in  their  approach.  They  need  to  stay  focused  on  the  end-result  and  the  path  towards 
that.  However,  they  also  need  to  be  creative  in  thinking  through  the  problems  at  hand  and 
arriving  at  solutions. 
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6. 2. 1.1.2  Effective  Communication 

A  large  part  of  being  a  systems  engineer  is  to  be  an  effective  communicator,  in  many  different  ways  as 
elaborated  below.  Since  systems  engineers  typically  have  a  broader  role  in  an  organization  than  those 
with  a  niche,  social  skills  in  general  and  communication  skills  in  particular,  are  very  important. 

•  Modes  of  Communication.  Systems  engineers  need  to  communicate  in  a  variety  of  modes  and 
types.  They  need  to  communicate  orally  (presentations,  discussion,  etc.)  and  in  writing 
(documents,  notes,  reports,  etc.).  They  must  be  good  speakers  and  also  be  good  listeners. 

•  Audience.  Systems  engineers  also  need  to  communicate  with  different  people  ranging  from 
technical  experts  in  the  "solution  domain"  and  various  stakeholders  in  the  "problem  domain".  A 
systems  engineer  serves  as  a  bridge  between  these  two  domains.  They  need  to  communicate 
with  individuals  as  well  as  teams.  As  elaborated  by  an  interviewee,  "you  have  to  look  high,  you 
have  to  look  low;  and  depending  on  your  audience  you  have  to  adjust  your  communication." 

•  Content.  The  content  of  their  communication  could  be  social,  managerial,  or  technical. 

•  Purpose.  A  systems  engineer  communicates  for  a  number  of  reasons,  most  commonly: 
understanding  needs,  negotiation,  information  brokering,  technical  arbitration,  driving 
consensus. 

6. 2. 1.1. 3  Flexible  Comfort  Zone 

Every  individual  typically  operates  within  a  "comfort  zone",  and  so  does  a  systems  engineer.  However, 
an  effective  systems  engineer  is  comfortable  to  go  outside  that  "comfort  zone"  where  required.  An 
interviewee  elaborated  "experts  have  the  fear  of  going  outside  of  their  circle  or  comfort  zone  that  keeps 
them  in  that  role  [of  an  expert]". 

•  Open  Minded.  An  effective  systems  engineer  is  open  to  different  views,  perspectives,  and 
opinions  from  others.  They  tend  to  be  unbiased  and  have  small  egos.  Systems  engineers  may 
have  to  re-look  at  a  decision  that  was  taken  many  years  ago  because  that  was  right  at  the  time 
of  decision-making  may  no  longer  be  so,  due  to  changes  in  technology  or  other  reasons. 

•  Rational  Risk  Taking.  Systems  engineers  are  required  to  take  rational  risks  while  dealing  with 
situations,  for  the  sake  of  the  larger  system.  Staying  within  one's  comfort  zone  for  fear  of  the 
unknown  may  not  help  in  engineering  the  optimal  system  that  meets  the  needs  of  the 
customer. 

•  Multidisciplinary.  Unlike  a  disciplinary  expert,  a  systems  engineer  needs  to  be  multidisciplinary. 
He  or  she  needs  to  understand  unfamiliar  disciplines  for  the  sake  of  the  system. 

•  Enjoys  Challenges.  A  systems  engineer  does  not  shy  away  from  challenges  but  rather  embraces 
them  as  opportunities. 

6. 2. 1.1.4  Smart  Leadership 

An  interviewee  said,  "Systems  engineers  gravitate  towards  leadership  because  of  the  larger  picture". 
Hence,  irrespective  of  where  they  are  in  an  organizational  hierarchy,  they  tend  play  a  leadership  role 
because  they  interact  with  many  individuals  and  teams,  and  are  focused  on  the  overall  system.  It  was 
also  said  "  a  good  systems  engineer  is  one  where  everyone  else  is  doing  well,  even  if  the  system 
engineer  is  struggling". 
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•  Quick  Learning  and  Abstraction.  Systems  engineers  tend  to  be  quick  learners,  and  have  the 
"intellectual  horsepower"  to  grasp  how  various  things  integrate  together.  They  also  have  the 
ability  to  abstract  what  is  said  and  capture  the  important  and  critical  bits  of  information. 

•  Knowing  when  to  stop.  An  effective  systems  engineer  knows  "what  is  good  and  good  enough". 
They  know  when  to  stop  doing  a  trade  and  move  on  without  spending  time  and  effort  to  arrive 
at  the  "absolutely  optimal  answer."  They  are  also  "willing  not  to  dive  into  the  detail".  In  dealing 
with  disciplinary  experts  in  the  team,  they  also  know  when  to  stop  being  a  leader,  to  offer  the 
space  and  respect  for  those  experts. 

•  Focused  on  'Vision'  for  System.  An  effective  systems  engineer  stays  focused  on  the  vision  or 
end-result  for  the  system.  This  focus  helps  the  systems  engineer  in  decision-making,  resolving 
conflict,  and  in  reducing  ambiguity. 

•  Ability  to  Connect  the  Dots.  A  systems  engineer  has  the  ability  to  "connect  the  dots"  from 
individual  perspectives  or  contributions  to  form  the  big  picture.  This  perspective  also  helps 
systems  engineers  spot  pitfalls,  identify  problems,  or  recognize  opportunities  that  are  not  seen 
by  others. 

•  Patience.  Though  in  a  leadership  role  where  decisions  need  to  be  taken,  often  quickly  and 
effectively  in  response  program  pressures,  systems  engineers  need  to  be  patient  and  not  jump 
to  any  conclusions.  They  demonstrate  structured  decision  making,  which  requires  patience  and 
discipline. 

6. 2. 1.1. 5  Self  Starter 

Systems  engineers  possess  a  lot  of  energy  and  drive  within  themselves,  without  relying  on  external 
factors  to  initiate  any  action  or  response. 

•  Curiosity.  Systems  engineers  tend  to  be  curious  and  have  a  "somewhat  wandering  mind".  They 
are  not  content  with  where  they  are  and  what  they  are  doing.  This  is  in  contrast  to  many  who 
just  say,  "this  is  my  job".  Systems  engineers  tend  to  ask  the  "why"  question  more  often,  without 
hesitating  to  wonder  if  it  is  a  "stupid"  question.  One  interviewee  said,  "I've  made  a  career  being 
the  stupidest  person  in  the  room." 

•  Passionate  and  Motivated.  Systems  engineers  tend  to  be  passionate  about  their  jobs  and  the 
systems  they  are  working  on.  They  possess  enough  motivation  within  them  to  drive  themselves 
and  others  in  the  team. 

•  Eager  to  Learn.  Systems  engineers  are  always  eager  to  learn  new  things  and  to  educate 
themselves.  In  most  cases,  they  are  willing  to  go  in  search  of  the  information  that  is  not  readily 
available.  They  also  recognize  that  things  are  constantly  changing  (technology,  needs,  products, 
processes,  etc.)  and  one  needs  to  stay  abreast  of  these. 

6.2. 1.2  Technical  Competencies 

The  question  of  required  competencies  for  systems  engineers  has  been  a  subject  of  inquiry  within  the 
community  for  the  last  several  years.  Recently,  the  International  Council  on  Systems  Engineering 
(INCOSE)  adopted  the  Systems  Engineering  Competencies  Framework,  which  outlines  required 
competencies  in  terms  of  systems  thinking,  holistic  lifecycle  view,  and  systems  engineering  management 
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(INCOSE  2010).  However,  there  is  still  a  debate  about  the  level  of  general  or  specialty  engineering 
competency  required  to  perform  systems  engineering  activities. 

The  Graduate  Reference  Curriculum  for  Systems  Engineering  (GRCSE)  states  that  systems  engineering 
masters  students  should  have  an  undergraduate  degree  in  engineering,  the  natural  sciences, 
mathematics,  or  computer  science;  at  least  two  years  of  practical  experience  in  some  aspect  of  SE;  and 
the  ability  to  effectively  communicate  technical  information  (Pyster  and  Olwell  (eds.)  2012).  These 
requirements  indicate  a  view  that  there  is  some  level  of  applied  engineering  understanding  required  for 
systems  engineers  to  be  effective.  However,  what  depth  of  engineering  competency  -  e.g.  in  electrical, 
mechanical,  civil  engineering,  etc.  -  that  is  required  is  heavily  debated.  Some  individuals  believe  -  and  in 
fact  some  organizations'  selection  processes  indicate  a  belief  -  that  systems  engineers  are  the  'best  of 
the  best'  from  other  engineering  disciplines.  (Pyster  and  Olwell  (eds.)  2013;  Adcock  et  al.  2012;  Ferris  et 
al.  2011) 

During  Helix  interviews,  when  participants  were  asked  about  the  need  for  and  importance  of  "technical 
competencies",  they  referred  to  two  types  of  competencies: 

•  Engineering  Competencies.  The  current  focus  of  Helix  and  the  organizations  that  participated 
were  all  "engineering"  systems.  And  so,  when  participants  referred  to  "technical  competencies", 
they  were  related  to  foundations  of  mathematics,  physics,  and  basic  engineering;  and 
specializations  in  specific  disciplines  such  as  electrical  engineering,  mechanical  engineering,  or 
aeronautical  engineering. 

•  Systems  Engineering  Competencies.  While  referring  to  "technical  competencies",  some 
interviewees  related  them  to  competencies  of  the  discipline  of  systems  engineering  such  as 
requirements,  life-cycle  processes,  and  architecting.  One  interviewee  said,  "not  all  systems 
engineers  will  be  leaders;  some  systems  engineers  will  stay  technical  and  do  architecture  and 
requirements  and  they  are  good  at  it." 

In  elaborating  the  importance  of  effective  systems  engineers  having  to  possess  these  technical 
competencies,  responses  varied  from  "not  important"  to  "very  important".  It  must  be  noted  that  these 
responses  are  based  on  the  personal  experiences  of  the  individuals  and  their  observations  of  "best" 
systems  engineers  they  have  encountered. 

In  analyzing  the  responses,  the  level  of  technical  competence  required  for  a  systems  engineer  may  be 
classified  into  two  areas: 

1.  At  Present:  More  Breadth  than  Depth 

Many  responses  referred  to  the  need  for  technical  competencies  currently  by  systems 
engineers.  Here  is  where  a  systems  engineer  is  expected  to  have  breadth  of  knowledge  and 
understanding  in  a  variety  of  disciplines  related  to  the  system.  Interviewees  explained  the 
reasons  why  systems  engineers  must  possess  this  breadth  of  technical  competence: 

o  To  be  familiar  with  technical  language; 
o  To  appreciate  the  expertise  and  value  of  technical  experts; 
o  To  understand  the  various  disciplines  related  to  the  system; 

o  To  understand  the  needs  of  the  customers  and  constraints  of  the  disciplinary  experts, 
and  to  evaluate  technical  feasibility; 
o  To  be  able  to  integrate  various  disciplines; 
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o  For  technical  communication  and  to  represent  views  of  disciplinary  experts  to 
stakeholders;  and 

o  To  be  able  to  bridge  the  problem  domain  and  solution  domain. 

2.  In  the  Past:  Depth  in  One  (or  more)  Disciplines 

While  providing  responses  related  to  the  importance  of  technical  competencies  for  systems 
engineers,  it  was  observed  that  interviewees  were  responding  to  technical  competencies  that 
systems  engineers  should  have  possessed  in  the  past,  but  which  they  may  not  be  actively  using 
today.  However,  those  technical  competencies  were  needed,  for  the  following  reasons: 

o  To  appreciate  the  value  of  disciplinary  analysis  and  design,  and  to  understand  the  time, 
effort,  and  resources  required; 

o  To  evaluate  the  validity  of  responses  provided  by  disciplinary  experts; 
o  To  appreciate  aspects  of  sub-system  level  optimization  and  the  need  for  system  level 
optimization; 

o  To  understand  the  perspectives  and  constraints  of  disciplinary  experts,  and  to  be  able  to 
properly  communicate  them  to  managers  and  other  non-technical  personnel  within  the 
organization;  and  to  the  customers  and  other  stakeholders; 
o  To  field  questions  from  stakeholders  in  the  absence  of  technical  experts;  and 
o  For  credibility  and  respect  within  the  team  and  among  stakeholders. 

It  is  implicit  in  the  above  discussion  that  typically  there  is  a  transition  from  an  engineer  being  a 
"specialty  engineer"  to  becoming  a  "systems  engineer",  and  this  can  happen  at  any  point  in  the  career 
path  of  the  individual.  In  this  transition,  it  is  important  for  the  individual  to  focus  on  breadth  rather  than 
depth,  which  may  mean  that  the  individual  needs  to  be  "willing  not  to  dive  into  the  detail",  as  discussed 
in  the  previous  subsection  on  Characteristics  (see  discussion  above).  Responses  indicated  that  many 
disciplinary  experts  sometimes  fail  to  make  this  transition  well,  which  can  make  them  ineffective  as 
systems  engineers.  One  interviewee  said,  "I  have  seen  people  who  have  come  from  [a]  very  strong 
physics  background,  very  strong  electrical  engineering  background  and  made  really  horrible  systems 
engineers."  For  those  disciplinary  experts  who  are  unable  to  make  this  transition  to  becoming  a  systems 
engineer,  "high  technical  skills  may  be  an  impediment"  since  "they  want  to  sub-optimize  and  nail  down 
all  the  requirements  and  nail  down  the  system". 

To  be  an  effective  systems  engineer,  there  is  a  level  of  technical  expertise  that  is  currently  required,  but 
more  important  is  to  be  able  to  transition  from  being  a  disciplinary  expert  to  being  a  systems  engineer 
possessing  the  required  technical  competencies  at  the  required  level  and  also  possessing  the 
characteristics  and  "soft  skills"  required  to  be  one.  One  interviewee  stated  that  the  rule  of  thumb  is  "you 
have  to  know  10%  depth  in  every  discipline  you  work  with". 


6.2,2,  The  Greatest  Contributions  of  Systems  Engineers 

Systems  engineering  as  a  discipline  will  only  prosper  if  its  practitioners  provide  value-added  to  the 
projects  and  programs  on  which  they  work.  Initially,  the  Helix  team  did  not  ask  explicit  questions  on  the 
benefits  that  each  individual  feels  his  or  her  SE  activities  provide.  However,  this  theme  appeared 
throughout  the  interviews.  The  most  common  interview  responses  regarding  the  benefits  of  systems 
engineers'  work  include: 

•  Translating  highly  technical  information  from  subject  matter  experts  (SMEs)  into  common 
language  that  other  stakeholders  can  understand; 
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•  Balancing  traditional  project  management  concerns  of  cost  and  schedule  with  technical 
requirements; 

•  Asking  the  right  questions; 

•  Seeing  relationships  between  the  disciplines; 

•  Staying  "above  the  noise"  and  identifying  pitfalls; 

•  Managing  emergence  in  both  the  project  and  the  system; 

•  Projecting  into  the  future;  and 

•  Getting  the  "true"  requirements  from  the  customer. 

Managing  emergence  in  both  the  project  and  in  the  system  being  developed  is  seen  as  an  interesting 
response;  however,  it  should  be  noted  that  it  was  not  stated  as  frequently  or  by  as  diverse  a  set  as  the 
other  items  listed  here. 

Translating  highly  technical  information  into  common  language  limits  the  miscommunications  and  often 
improves  stakeholder  satisfaction,  as  they  can  understand  why  certain  desires  may  not  be  met  or  why 
certain  project  aspects  may  take  longer  than  others. 

Several  interviewees  reported  that  the  specialty  engineers  they  work  with  often  feel  so  over-burdened 
by  cost  and  schedule  pressures  that  it  is  difficult  for  them  to  meet  the  bare  minimum  technical 
requirements.  Systems  engineers  "protect  their  teams"  by  helping  project  or  program  managers 
understand  the  technical  issues,  and  therefore  support  decisions  about  when  cost  and  schedule  changes 
are  absolutely  required  to  meet  technical  requirements.  Conversely,  systems  engineers  can  also  help 
determine  when  technical  concerns  are  not  required  but  are  rather  "polishing  the  apple"  and  can  be 
traded  in  favor  of  cost  and  schedule. 

Several  interviewees  indicated  that  this  also  allows  them  to  address  potential  problems  before  they 
have  a  chance  to  impact  the  system  as  a  whole;  i.e.  it's  an  effective  tool  for  risk  management  that  allows 
examination  of  the  approach  overall,  not  just  the  specifics  of  one  component  or  discipline. 

As  one  interviewee  stated,  this  allows  "everybody  [to]  get  to  the  same  conclusion  that  this  is  the  correct 
path  or  the  best  path  to  follow  because  you  drew  it  out  into  the  open".  In  other  words,  systems 
engineers  help  to  balance  competing  technical  approaches  by  addressing  system-wide  implications  of 
those  approaches.  Bringing  this  perspective  to  the  team  enables  the  technical  specialists  to  see  where 
"it  might  have  been  better  actually  if  [they]  had  come  up  with  a  suboptimal  solution  for  [their]  little  part 
because  you'd  arrive  at  a  much  better  solution  for  the  collective." 

There  is  a  clear  relatinship  between  the  contributions  of  systems  engineers  and  the  skills  required  to 
bring  those  contributions  to  fruition.  In  future  reports,  the  team  will  explore  the  relationships  between 
these  elements  as  it  works  to  build  a  theory  of  systems  engineers. 


6.2.3.  What  Makes  Systems  Engineers  Most  Effective 

The  underlying  intent  of  all  workforce  development  initiatives  is  to  create  or  mature  an  effective 
workforce.  Therefore  understanding  what  increases  or  decreases  a  systems  engineer's  ability  to  be 
effective  is  of  great  importance  for  anyone  engaging  in  such  workforce  development  activities.  This 
section  will  explore  what  effectiveness  means  for  a  systems  engineer  and  what  enables  effectiveness. 
Inhibitors  of  effectiveness  can  be  found  in  the  next  section. 

6.2.3.1  Effectiveness 

In  order  to  explain  what  best  enables  a  systems  engineer  to  be  effective,  it  is  first  important  to  define 
effectiveness.  The  Helix  project  began  with  a  generic  definition  of  effectiveness,  which  was  that  "an 


December,  2013 


Helix  Technical  Report 


31 


individual  systems  engineer  is  effective  when  the  outcomes  for  which  he  or  she  is  individually 
responsible  are  achieved  as  a  result  of  the  systems  engineering  activities  he/she  performs."  This  generic 
definition  was  presented  to  interviewees,  who  were  asked  to  respond  to  and  add  to  it.  In  general, 
interviewees  agreed  with  this  as  a  generic  outline. 

In  more  specific  terms,  interviewees  indicated  that  a  systems  engineer  is  effective  when  he  or  she 
delivers  a  system  that  aligns  with  schedule,  cost,  and  technical  requirements,  and  satisfies  the  customer, 
"getting  quality  equipment  out  to  the  field".  Metrics  for  evaluating  the  effectiveness  of  systems 
engineers  are  difficult  for  many  reasons;  specifically,  interviewees  believed  that  the  time  lag  between  SE 
decisions  and  the  results  of  those  decisions  being  seen  made  any  quantitative  assessment  particularly 
difficult. 

Though  metrics  for  the  effectiveness  of  systems  engineers  seem  to  be  uncommon  and  imprecise, 
several  qualitative  indicators  were  discussed  by  the  interviewees.  For  example,  "you  haven't  had  a 
major  rework  because  of  a  requirement  that  was  missed  or  an  interface  that  was  missed";  systems 
engineers  have  "...  anticipate[d]  where  the  issues  might  be  ..."  and  developed  methods  to  mitigate  or 
resolve  issues  quickly.  One  participant  stated  that,  "an  indirect  method  to  measure  effectiveness  is  to 
look  at  the  number  of  programs  the  systems  engineer  has  participated  in,  non-conformances,  red 
programs,  win  rates."  Finally,  "in  many  ways  a  systems  engineer  is  an  orchestrator  for  everybody  else  to 
do  their  job.  Whether  or  not  you're  writing  requirements  or  you're  conceiving  what  the  whole  system  is, 
it's  an  orchestration  of  everyone.  So  [an  effective]  system  engineer  is  one  where  everyone  else  is  doing 
well.  If  the  system  engineer  is  struggling  but  everybody  else  is  doing  well,  he's  probably  struggling  the 
right  way." 

With  this  view  of  effectiveness  in  mind,  each  interviewee  was  asked  to  identify  factors  that  make  it 
easier  for  them  to  be  effective.  There  was  a  wide  variety  of  answers,  some  of  which  overlap  with  some 
of  the  characteristics  of  systems  engineers  discussed  earlier.  This  report  explores  the  factors  of  diversity 
of  experiences  and  mentoring,  as  from  the  interviews  conducted  thus  far  it  is  evident  that  these  are  two 
of  the  most  important  aspects  of  enabling  effective  systems  engineers.  This  section  will  also  discuss  the 
relationship  between  the  organizational  value  of  systems  engineering  and  effectiveness. 

6.2. 3.2  Diverse  Experiences 

SE  is  a  practical  discipline;  its  benefits  are  conferred  primarily  through  application.  It  is  widely  accepted 
that  practical  experiences  are  therefore  a  critical  component  of  the  development  of  systems  engineers. 
There  is  much  debate  in  the  community  about  how  much  time  is  required  to  mature  a  systems  engineer 
and  what  types  of  experiences  are  required.  This  report  does  not  attempt  to  address  all  concerns  with 
regards  to  experience;  this  will  be  a  major  focus  for  Helix  going  forward.  However,  a  consistent  theme 
from  almost  all  interviewees  was  that  diversity  of  experience  is  critical  to  the  maturation  of  systems 
engineers. 

Specifically,  the  majority  of  interviewees  believe  that  systems  engineers  need  to  gain  multiple  types  of 
experiences,  which  may  include: 

•  Different  parts  of  the  SE  life  cycle; 

•  Different  types  of  life  cycles,  e.g. 

o  Quick  Reaction  Capability  (QRC); 

o  More  formal  acquisition  life  cycles,  aligning  with  DoD  5000.2;  and 
o  Internal  research  and  development  (IR&D)  projects; 

•  Different  aspects  of  a  system  (part,  component,  subsystem,  system);  and 

•  Different  critical  orthogonal  attributes  of  the  system  (e.g.  weight,  size,  etc.). 
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This  diversity  of  experiences  enables  systems  engineers  to  develop  the  characteristics  and  competencies 
that  make  them  more  effective,  as  discussed  above.  For  example,  seeing  in  practice  the  interaction 
between  different  disciplines,  the  results  of  decisions  made  in  one  phase  of  the  life  cycle  manifested  in 
another  phase  of  the  life  cycle,  and  experiencing  problems  with  interfaces  all  help  a  systems  engineer  to 
develop  and  refine  big  picture  thinking. 

This  diversity  of  experiences  also  enables  systems  engineers  to  develop  pattern  recognition  across 
domains  that  help  them  to  identify  risks  or  issues  early  ("give  red  flags")  or  identify  "golden 
opportunities"  that  might  have  otherwise  been  missed.  These  abilities  are  supported  by  the 
development  of  the  habit  of  asking  "the  why  questions";  i.e.  as  systems  engineers  experience  different 
approaches  in  different  contexts,  they  have  the  opportunity  to  ask  and  learn  why  one  approach  may  be 
better  in  one  context  than  another.  This  in  turn  gives  them  the  ability  to  use  that  context  to  examine 
approaches  in  new  areas  and  probe  when  an  approach  does  not  seem  to  fit  the  context.  From  this, 
systems  engineers  build  their  "toolbox"  of  approaches,  methods,  and  specific  tools,  which  enable  them 
to  perform  SE  tasks  more  efficiently  in  future.  Some  interviewees  have  also  stated  that  more  diverse 
experiences  correlate  with  having  more  open-minded  approaches  and  therefore  better  enable  SE. 

Further  benefits  of  diversity  of  experience  discussed  during  the  interviews  included  gaining  the 
opportunities  to  experience  failure  and  learn  from  it  ("build  scar  tissue").  It  is  important  to  note  this  scar 
tissue  can  occur  even  without  diverse  experiences;  however,  gaining  scar  tissue  across  multiple  areas  is 
seen  as  a  means  to  better  enable  the  internalization  of  systems  engineering  principles.  For  example, 
some  interviewees  stated  that  individuals  who  had  worked  only  on  QRC-type  projects  did  not 
understand  the  value  of  formal  requirements  generation.  They  needed  the  opportunity  to  fail  (on  a 
small  scale)  because  of  incorrect  requirements  on  a  more  formal  project  in  order  to  begin  building  the 
skills  to  be  more  discerning  about  the  application  of  the  systems  engineering  process  in  specific 
contexts.  Some  interviewees  also  stated  that  experience  with  different  program  mangers  -  some  who 
are  supportive  of  SE  and  others  who  want  to  jump  immediately  into  design  -  provides  more 
opportunities  to  experience  rework,  and  therefore  reinforce  the  SE  approach. 

Finally,  there  are  orthogonal  aspects  of  systems  that  are  of  particular  importance.  For  example,  in  many 
defense  systems,  weight  is  a  critical  system  attribute  that  impacts  all  components,  overall  system 
performance,  and  the  ability  of  the  system  to  fulfill  the  concept  of  operation.  The  critical  attributes  may 
differ  depending  on  the  domain,  but  it  appears  that  for  each  organization  there  is  at  least  one.  Several 
interviewees  stated  that  if  they  had  understood  the  importance  and  impact  of  these  system  aspects 
earlier,  it  would  have  helped  them  create  better  designs  with  fewer  errors.  They  also  stated  that  the 
only  way  they  understood  this  was  to  see  the  implications  in  practice;  this  is  not  something  that  they 
believed  they  could  fully  grasp  from  a  training  course.  It  may  be  beneficial  for  systems  engineering 
organizations  to  identify  these  critical  systems  attributes  for  the  relevant  domain(s)  and  ensure  that 
junior  systems  engineers  gain  experience  and  focus  on  these  early  in  their  careers. 

Many  organizations,  including  several  of  the  organizations  participating  in  Flelix,  have  programs 
designed  to  enable  systems  engineers  to  build  a  portfolio  of  experiences  early  in  their  careers.  In 
addition  to  having  the  benefits  listed  above,  as  a  workforce  development  tool,  interviewees  indicated 
that  this  approach  enables  individuals  to  select  which  areas  of  systems  engineering  best  align  with  their 
skill  sets  and  interests.  Several  interviewees  stated  that  by  exposing  junior  systems  engineers  to  the 
different  options,  they  were  more  likely  to  choose  an  area  they  would  be  interested  in  and  less  likely  to 
leave  the  company  or  transfer  to  a  non-technical  track. 

There  is  a  caution  about  diversity  of  experiences,  however:  diversity  must  be  balanced  with  depth. 
Several  interviewees  stated  that  they  have  observed  a  recent  trend  in  younger  systems  engineers  that  is 
overly  focused  on  breadth  -  and  that  these  'generalist'  systems  engineers  struggle  without  the  proper 
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engineering  underpinnings.  A  few  ways  to  deal  with  this  include  allowing  systems  engineers  to  gain 
experience  throughout  the  entire  life  cycle  on  a  single  project.  This  allows  breadth  of  experience  in  the 
life  cycle,  but  depth  in  the  domain.  Another  challenge  with  some  rotational  programs  is  that  diverse 
experiences  are  gained  through  short  assignments.  In  these  instances,  there  is  no  opportunity  to  "break 
something"  or  to  see  the  longer-term  effects  of  one's  efforts.  A  commonly  cited  approach  that 
interviewees  indicates  works  well  is  diversity  early  in  ones  career  but  with  some  stable  aspect  (e.g. 
project,  domain,  etc.),  a  deeper  dive  with  less  diversity  mid-career,  and  diversity  at  a  higher  level  later 
career  with  the  deep  technical  experience  to  support  higher-level  positions. 

The  question  of  how  long  it  takes  to  mature  a  systems  engineer,  the  optimal  balance  between  diversity 
of  experiences  and  depth  in  a  domain  or  specialty,  and  the  best  timing  for  these  experiences  are  still 
debated,  and  the  Helix  team  will  continue  to  investigate  these  issues. 

6.2. 3.3  Mentoring 

As  discussed  above,  diverse  experiences  are  critical  for  the  development  of  systems  engineers. 
Interviewees  see  mentoring  as  a  strong  accelerating  function  for  growing  the  competencies^  that  are 
required  to  be  an  effective  systems  engineer.  Mentoring,  then,  is  a  tool  that  enables  systems  engineers 
to  get  more  value  from  their  experiences. 

Mentoring  has  had  several  meanings  to  the  individuals  interviewed  thus  far:  career  mentorship,  which 
focuses  on  helping  individuals  understand  how  they  can  best  grow  as  systems  engineers;  organizational 
mentorship,  which  focuses  on  helping  an  individual  navigate  within  his/her  current  organizational 
environment;  and  technical  mentorship,  which  focuses  on  helping  an  individual  improve  his/her 
technical  performance  in  context.  Technical  mentorship  often  involves  the  mentor  acting  as  SME  and 
helping  a  less  experienced  person  identify  pitfalls  and  opportunities  and  is  generally  discussed  similarly 
to  an  apprenticeship. 

In  most  formal  mentoring  arrangements  discussed,  the  mentor  and  mentee  are  paired  up  by  the 
organization.  This  does  not  always  work  for  two  main  reasons:  "not  every  senior  systems  engineer  is  a 
good  mentor"  and  a  good  match  between  the  mentor  and  mentee  cannot  always  be  guaranteed.  There 
is  no  defined  way  of  becoming  a  good  mentor.  Interviewees  were  not  able  to  identify  any  training  that 
could  be  provided.  However,  there  are  indicators  to  identifying  a  good  mentor  that  organizations  can 
use  to  select  mentors  for  a  formal  mentoring  initiative,  many  of  which  include  the  traits  discussed  above 
such  as  big  picture  thinking,  staying  "above  the  noise",  and  the  ability  to  communicate  technical 
information.  These  skills  allow  a  mentor  to  help  his/her  mentee  see  through  to  the  true  issues  in  their 
systems  and  determine  appropriate  solutions;  ideally,  these  solutions  will  be  applied  and  the  mentee 
can  then  understand  the  outcomes  in  practice.  This  sort  of  apprenticeship  was  cited  by  many  as  crticial 
to  the  development  of  good  systems  engineers.  The  preponderance  of  interviewees  stated  that  they 
participate  in  some  form  of  mentoring,  though  most  of  them  stated  that  the  more  successful  mentoring 
experiences  have  been  informal. 

It  is  worth  noting  that  the  above  are  all  forms  of  active  mentorship  -  an  interaction  between  individuals 
to  enable  growth.  There  were  also  many  examples  of  passive  mentorship  mentioned  -  i.e.  many 
interviewees  stated  that  having  strong  systems  engineers  available  as  role  models  was  also  very 
important  for  their  growth.  Interviewees  stated  that  this  is  facilitated  when  senior  systems  engineers 
are  "doers",  individuals  who  "roll  up  their  sleeves"  to  provide  direct  technical  value  on  a  system. 


^  The  data  for  Helix  shows  that  there  are  different  types  of  competencies:  general  engineering  competencies;  specialty  engineering 
competencies;  systems  engineering  competencies;  and  non-technical  competencies  such  as  leadership,  communication,  etc. 
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The  absence  of  mentoring  may  not  be  apparent  in  any  direct,  tangible  manner.  There  may  be  other 
ways  through  which  systems  engineers  may  obtain  the  competencies  fostered  by  a  mentorship  or 
apprenticeship-style  arrangement.  However,  interviewees  indicated  that  these  alternative  routes  might 
be  more  laborious,  time  consuming,  or  less  certain  than  mentoring.  The  Helix  team  will  explore  this 
question  more  deeply  in  future  interviews. 

6.2. 3.4  Value  of  Systems  Engineering 

Though  not  surprising,  several  interviewees  stated  that  when  systems  engineering  is  valued  as  a 
discipline,  it  creates  an  environment  in  which  it  is  easier  to  apply  systems  engineering  principles  and 
processes.  This  can  occur  when  a  customer  places  particular  value  on  systems  engineering  or  when  an 
organization  places  value  on  the  discipline.  Interviewees  stated  that  when  systems  engineering  is 
desired  by  the  customer,  it  becomes  very  easy  to  implement  the  process,  as  most  organizations 
participating  in  Helix  are  driven  by  a  desire  to  "make  the  customer  happy". 

This  desire  for  SE  may  be  counterbalanced  by  budget  and  schedule  considerations.  (See  discussions 
below.) 


6.2,4,  What  Makes  Systems  Engineers  Least  Effective 

The  Helix  team  has  observed  that  although  the  systems  engineers  interviewed  believe  that  they  make 
valuable  contributions  to  their  projects  and  programs,  there  are  factors  that  seem  to  inhibit  their  ability 
to  be  as  effective  as  they  would  like.  Some  of  these  factors  are  the  inverse  of  the  enabling  factors 
discussed  above;  e.g.  if  a  mentoring  culture  or  having  access  to  strong  mentors  is  an  enabler,  then  lack 
of  mentorship  opportunities  may  inhibit  effectiveness.  Likewise,  several  factors  discussed  have  a 
correlation  with  a  lack  of  the  characteristics  of  effective  systems  engineers  as  discussed  above;  e.g. 
effective  systems  engineers  are  holistic  thinkers,  therefore  systems  engineers  who  have  trouble  thinking 
outside  of  specific  disciplines  will  be  less  effective.  While  these  types  of  statements  are  consistent  with 
the  data,  this  section  focuses  on  interviewee's  statements  about  inhibitors  that  are  not  a  simple 
corollary  to  the  enabling  factors. 

6.2.4.1  Ambiguous  Definition  of  Systems  Engineering 

The  majority  of  interviewees  across  several  organizations  stated  that  the  organizational  understanding 
of  "systems  engineering"  is  blurry.  This  may  be  for  a  variety  of  reasons.  Some  of  the  organizations 
interviewed  to  date  do  not  have  a  formal  definition  of  systems  engineering;  clearly  if  there  is  no 
standard  definition,  there  will  be  inconsistent  views  on  what  systems  engineering  is  and  therefore  what 
systems  engineers  do.  Of  the  organizations  that  do  have  a  formal  definition,  the  definition  was  often  not 
well  known  -  even  to  the  systems  engineers.  Another  commonly-cited  problem  is  dilution  of  the  term  - 
using  the  term  "systems  engineering"  to  cover  too  many  activities,  some  of  which  have  been  labeled 
differently  for  years.  Some  interviewees  observed  that  this  was  due  to  organizational  recognition  of  the 
importance  of  SE,  but  the  implementation  has  made  it  less  understandable.  One  interviewee  stated  that 
the  shift  occurred  while  he  was  undergoing  project  management  training  and  "suddenly  everything  that 
[he]  thought  was  project  management  was  systems  engineering."  This  is  true  also  for  SE  terminology:  "... 
So  when  I  say  'subsystems'  they  don't  mean  the  subsystems  I'm  talking  about.  When  they  say  system, 
they  don't  mean  the  system  I'm  talking  about." 

6.2A.2  Unclear  Use  of  ‘Systems  Engineer’  Title 

Confusion  about  the  term  "systems  engineering"  is  further  compounded  by  the  fact  that  the  title 
'systems  engineer'  is  used  differently  within  each  participating  organization.  In  some  organizations. 
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there  may  be  a  'systems  engineering  division'  or  similar  and  all  individuals  within  that  organization  are 
labeled  'systems  engineers',  even  if  they  actually  perform  electrical  engineering,  or  manage  logistics, 
etc.  Several  interviewees  stated  that  this  makes  it  very  hard  to  make  a  case  for  the  value  of  SE  because 
their  peers  do  not  understand  what  they  will  get  when  they  add  a  "systems  engineer"  to  the  project.  In 
organizations  where  the  label  of  "systems  engineer"  is  applied  more  intentionally,  this  is  less  of  an  issue, 
though  interviewees  from  these  organizations  still  indicate  problems  with  the  consistency  of 
terminology. 

6.2.4.3  Limited  Vaiue  of  Systems  Engineering  in  Organizationai  Cuiture 

As  seen  in  the  interview  data,  one  of  the  primary  activities  of  a  systems  engineer  is  often  to  explain  and 
get  peer  buy-in  on  the  value  of  SE.  When  there  is  no  consistency  in  the  use  of  this  terminology,  this  task 
is  more  difficult  and  individuals  must  be  won  over  on  a  case-by-case  basis.  This  is  particularly  challenging 
when  project  managers  do  not  understand  the  value  or  role  of  SE  with  regard  to  the  sub-optimization  of 
components  for  the  good  of  the  system;  i.e.,  project  managers  may  view  systems  engineers  as 
unnecessarily  limiting  the  effectiveness  of  specialty  engineers  if  they  do  not  understand  the  holistic 
perspective  of  SE.  Systems  engineers  must  ensure  that  both  the  project  managers  and  specialty 
engineers  understand  the  interplay  between  disciplines;  again,  this  is  more  difficult  when  the  cultural 
value  for  SE  is  weak.  This  tension  is  often  also  manifest  in  the  schedule;  if  project  managers  don't 
understand  the  value  of  performing  rigorous  requirements  development,  for  example,  they  often 
become  frustrated  with  systems  engineers  for  "holding  up  the  process".  As  one  interviewee  stated,  the 
"general  perception  [is]  that  SE  is  good  to  do  -  but  when  it  comes  time  to  fund  it,  it's  often  the  first  thing 
cut."  Systems  engineers,  therefore,  have  to  spend  more  time  and  energy  trying  to  convince  the 
organization  of  the  value  of  SE  than  would  their  counterparts  from  classical  engineering  disciplines. 

6.2.4.4  Systems  Engineering  Toois 

Many  individuals  highlighted  a  lack  of  SE  tools  as  making  it  more  difficult  for  them  to  do  their  work. 
Some  indicated  that  this  is  due  to  organizational  inhibitors,  so  that  the  tools  that  exist  are  not  available 
to  them.  As  discussed  above,  this  seems  to  be  linked  with  the  organizational  value  of  SE.  When  tools  are 
available,  several  interviewees  stated  that  there  is  insufficient  training  or  mentorship  on  the  tools,  which 
makes  it  difficult  for  them  to  be  effectively  used.  Some  interviewees  stated  that  the  tools  available  to 
them  are  simply  not  robust  enough  to  support  the  true  system-level  analysis  on  very  complex  systems. 
Finally,  there  is  a  feeling  that  a  lot  of  time  and  effort  are  required  to  convince  project  managers  of  why 
these  tools  should  be  used.  No  justification  is  required  for  tooling  for  other  disciplines  (e.g.  "a  designer 
doesn't  have  to  convince  the  PM  [program  manager]  that  he  needs  access  to  CADS  [software].").  It 
seems  that  it  is  simply  not  as  ingrained  in  corporate  cultures  that  SE  tools  are  needed. 

6.2.4.5  Visibility  of  Failures  vs.  Successes 

A  key  challenge  cited  in  championing  the  value  of  SE  is  that  failures  are  extremely  visible,  while 
successes  are  often  the  opposite;  i.e.  there  is  little  to  draw  attention  to  success.  Instead,  effective 
systems  engineers'  projects  are  marked  by  a  lack  of  rework  and  successful  fielding  -  i.e.,  what  is 
traditionally  expected.  Some  interviewees  stated  that  systems  engineers  could  take  advantage  of  this  by 
citing  examples  of  where  SE  efforts  failed  and  all  of  the  associated  effects.  They  could  indicate  where 
improved  SE  most  likely  would  have  led  to  a  better  outcome. 

6.2.4.6  Valuing  Process  over  Critical  Thinking 

Even  in  organizations  that  recognize  or  have  begun  to  recognize  the  value  of  systems  engineering,  the 
actual  implementation  of  SE  may  make  it  more  difficult  for  systems  engineers  to  perform  efficiently.  For 
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example,  some  systems  engineers  said  parts  of  their  organization  value  the  SE  process  over  critical 
thinking.  When  failures  happen,  the  SE  process  is  expanded  to  address  that  failure,  creating  a 
heavyweight  and  inflexible  process. 

Other  interviewees  stated  that  though  at  the  organizational  level  there  is  a  message  of  valuing  systems 
engineering,  it  often  becomes  a  "check  the  box"  process  -  engineers  begin  focusing  on  producing 
specific  deliverables  within  the  process  instead  of  using  those  deliverables  as  tools  to  improve  the 
system.  In  fact,  several  interviewees  stated  that  this  mentality  becomes  pervasive  over  time,  to  the 
point  that  even  if  systems  engineering  deficiencies  are  identified  in  a  review,  a  system  may  still  pass  the 
review,  with  no  additional  steps  or  rework  required. 

6.2.4.7  Younger  Systems  Engineers  Fail  to  Recognize  the  Importance  of  Process 

As  an  inverse  of  the  previous  discussion,  several  interviewees  stated  that  many  of  the  younger  systems 
engineers  have  grown  into  the  discipline  using  less  rigorous  processes.  For  example,  the  QRC  approach, 
used  heavily  in  the  defense  community  in  the  last  decade,  was  a  necessary  enabler  to  allow  fielding  of 
systems  to  the  warfighter  in  an  extremely  expedited  manner.  This  "quick  reaction"  has  been  enabled  by 
a  necessary  reduction  is  the  formality  of  the  systems  engineering  process.  As  Operations  Iraqi  Freedom 
and  Enduring  Freedom  draw  to  a  close,  the  perceived  need  for  the  QRC  approach  is  also  decreasing.  This 
means  that  the  more  formal  approach  of  traditional  systems  acquisition  may  be  implemented  more 
strongly  going  forward.  Several  examples  were  cited  of  systems  engineers  who  have  had  only  QRC 
experience  trying  to  transition  to  a  project  with  a  more  rigorous  SE  process.  These  younger  systems 
engineers  have  struggled,  often  expending  more  energy  fighting  against  the  process  than  working  with 
it. 

Several  interviewees  stated  that  it  is  important  to  tailor  the  systems  engineering  process  so  that  it  is 
appropriately  rigorous  for  the  type  of  system  being  designed.  However,  for  the  young  systems  engineers 
with  little  or  no  experience  in  more  formal  programs,  the  default  position  seems  to  be  "less  process  is 
better".  Specific  examples  were  discussed  when  individuals  who  performed  well  in  QRC  approaches  did 
very  poorly  in  larger,  more  traditional  programs.  It  was  stated  that  this  is  more  common  when  someone 
saw  only  successes  with  QRC  programs.  A  few  interviewees  stated  that  individuals  who  had  the 
opportunity  to  see  where  the  less  rigorous  QRC  process  allowed  a  breakdown  in  system  capability, 
integration,  etc.,  tended  to  be  more  open  to  rigorous  systems  engineering  approaches. 

6.2.4.8.  Inadequate  Knowledge  Management 

Several  interviewees  discussed  the  lack  of  and  need  for  knowledge  management  activities.  This  was 
generally  discussed  in  two  contexts.  First,  general  best  practice  of  identifying  lessons  learned  on 
programs  -  many  interviewees  stated  that  they  spend  a  large  amount  of  time  capturing  lessons  learned 
upon  completion  of  a  project  or  program.  However,  when  they  need  to  access  this  information  in  future, 
it  is  almost  impossible  to  find.  Second,  many  interviewees  stated  that  they  are  concerned  about  the  loss 
of  knowledge  as  more  senior  systems  engineers  leave  the  workforce  (for  more  information,  see 
subsection  6.2.6. 1,  below).  Many  of  these  people  believed  that  rigorous  knowledge  management  could 
help  prevent  or  mitigate  some  of  this  loss.  However,  several  organizations  have  no  organized  knowledge 
management  approach  and  of  the  ones  that  do  have  a  formal  approach,  several  interviewees  indicated 
that  there  is  a  breakdown  in  input,  storage,  and  search  capabilities,  rendering  these  sometimes  large- 
scale  efforts  almost  useless. 
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6.2.5  Perceived  Risks  to  the  Systems  Engineering  Workforce 


Nearly  every  interviewee  was  asked  to  identify  his  or  her  top  one  or  two  areas  of  concern  with  regard  to 
the  SE  workforce  over  the  next  five  years.  There  were  many  organization-specific  concerns,  but  across 
organizations,  several  patterns  emerged.  The  most  commonly  cited  risks  are: 

•  The  high  percentage  of  senior  systems  engineering  personnel  nearing  retirement; 

•  The  shifting  environment;  and 

•  The  expectations  of  the  junior  systems  engineering  workforce. 

Clearly,  concern  about  retirement  would  be  addressed  if  the  more  junior  workforce  was  adequate  to  fill 
any  gaps.  However,  there  are  also  concerns  about  the  expectations  -  and  resulting  retention  -  of  the 
younger  systems  engineering  workforce.  All  of  this  is  occurring  in  the  context  of  the  shifting 
environment,  which  includes  both  operational,  acquisition,  and  economical  concerns. 

Each  of  these  issues  is  discussed  in  more  detail  below. 

6.2.5. 1  High  Percentage  of  Senior  Systems  Engineers 

As  stated  in  the  Introduction  of  this  report,  one  area  of  interest  to  DoD  is  the  age  profile  of  the  SPRDE 
(Systems  Planning,  Research,  Development,  and  Engineering)  workforce,  specifically  the  implication  of 
the  high  percentage  of  the  workforce  at  or  nearing  retirement  age.  (Welby  2010,  Welby  2011,  Torelli 
2012)  This  profile  can  be  seen  in  Figure  13. 


Figure  14.  Age  Profile  of  the  DoD  SE  (SPRDE)  workforce  (Welby  2010) 


The  Helix  team  posed  this  issue  to  the  interviewees,  with  mixed  reactions.  First,  this  "bathtub  curve" 
profile  does  not  exist  in  all  organizations.  In  addition,  some  individuals  stated  that  they  felt  they  had 
sufficient  mechanisms  in  place  to  deal  with  this  and  it  was  not  a  concern  or  that  the  junior  systems 
engineering  workforce  is  extremely  promising  and  will  "rise  to  the  challenge".  The  current  sample  of 
seven  organizations  is  too  small  to  draw  conclusions  about  the  wider  systems  engineering  workforce. 
However,  it  is  evidence  that  this  may  not  be  a  consistent  problem. 

Second,  in  organizations  that  do  have  a  higher  percentage  of  at-  or  near-retirement  systems  engineers, 
there  were  very  mixed  reactions  to  the  issue.  One  consistent  issue,  however,  was  the  common  concern 
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for  losing  "specialty"  systems  engineers  -  those  with  deep  expertise  in  a  specialty  discipline  or  a  specific 
domain  or  application  -  over  "generalist"  systems  engineers. 

In  organizations  where  this  is  a  major  concern,  there  are  several  different  approaches  for  dealing  with 
this  issue.  For  example,  several  individuals  cited  a  formal  "succession  plan"  in  which  individuals  who 
may  soon  retire  are  identified  along  with  potential  successors.  The  methods  of  grooming  potential  new 
senior  systems  engineers  almost  all  include  mentoring  or  apprenticeship;  some  include  specific  types  of 
training.  The  level  at  which  this  is  handled  is  not  consistent;  in  some  organizations  it  is  handled  by  HR,  in 
others  by  managers,  and  in  others  it  is  owned  by  the  systems  engineers  themselves.  Some  organizations 
also  state  that  they  bring  back  senior  systems  engineers  as  consultants  for  a  period  of  time  to  help  ease 
their  transitions. 

Several  individuals  stated  that  it  was  not  only  not  a  problem,  but  instead  an  opportunity.  As  one 
interviewee  put  it:  "Good  riddancel"  Though  flippant,  this  response  highlights  a  pattern  seen  in 
responses  from  several  mid-level  systems  engineers.  There  are  perceived  to  be  a  very  limited  number  of 
senior  systems  engineer  positions,  so  individuals  who  are  mid-career  in  systems  engineering  must  wait 
until  a  senior  position  is  open  before  they  can  advance  their  careers.  This  has  led  to  frustration,  so  the 
possibility  of  increased  retirement  rates  may  provide  opportunities  for  people  looking  to  advance. 

6.2.5.2  Shifting  Environment 

In  the  current  environment  -  a  shift  from  a  war-time  posture  to  a  peace-time  posture  -  interviewees 
expect  to  see  an  increase  in  measures  like  sequestration  along  with  the  continued  draw-down  of 
deployment  activities  and  a  decreased  need  for  QRC  developments.  Smaller  and  fewer  programs  are 
anticipated,  leaving  less  opportunity  for  diversity  of  experiences  and  making  it  more  difficult  for 
seasoned  senior  systems  engineers  to  develop.  Further  pressures  to  do  programs  more  cheaply  is 
anticipated  to  have  a  negative  effect  on  systems  engineers,  for  though  it  should  mean  a  focus  on  the 
value  add  of  systems  engineering,  interviewees  believe  that  instead  the  up-front  costs  will  become 
harder  to  justify. 

The  retirement  issue  may  also  be  complicated  by  early  retirement,  which  is  anticipated  in  the  new 
financial  and  defense  climate.  Several  interviewees  stated  that  the  shifting  environment  may  make  SE  as 
a  discipline  less  desirable.  Specifically,  there  is  concern  that  young  systems  engineers  may  not  believe 
that  SE  will  provide  sufficient  career  opportunities.  Several  interviewees  stated  that  they  have  already 
seen  a  trend  that  many  of  the  younger  systems  engineers  choose  to  go  into  management  roles  and 
quickly  move  out  of  technical  roles.  This  compounds  the  issue  of  being  able  to  back-fill  retirees. 

6.2.5. 3  Expectations  of  Young  Systems  Engineers 

Several  interviewees  expressed  concern  over  the  expectation  of  the  younger  generation  of  systems 
engineers.  Specifically,  several  interviewees  cited  grooming  programs  for  young  systems  engineers  that 
seem  to  send  the  message  that  these  high-potential  individuals  will  quickly  become  leaders  in  systems 
engineering.  There  is  often  an  expectation  that  in  a  short  period  of  time  (5-10  years  is  often  cited)  that 
these  younger  systems  engineers  will  be  in  senior  systems  engineering  positions.  As  stated  above,  the 
population  is  in  many  places  top-heavy,  with  few  open  senior  positions.  This  leads  to  disillusionment, 
causing  many  younger  systems  engineers  to  move  between  organizations  looking  for  upward  mobility. 
The  changing  environment,  as  discussed  above,  also  impacts  this  view. 

Several  interviewees  also  cited  examples  of  young  individuals  who  did  quickly  obtain  senior  positions, 
but  who  were  ultimately  not  prepared  for  the  task.  They  cited  lack  of  diversity  of  experiences  as  a 
primary  factor  that  individuals  often  do  not  thrive  in  these  positions.  Again,  these  perceived  failures 
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might  be  encouraging  the  younger  generation  of  systems  engineers  to  leave  the  technical  roles  and 
pursue  management  roles. 
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7.  Alignment  of  Findings  with  Helix  Research  Questions 


The  purpose  of  the  Helix  project,  as  discussed  in  Section  2,  is  to  answer  three  research  questions.  While 
all  insights  into  the  systems  engineering  workforce  are  relevant,  it  is  critical  to  understand  how  the 
information  gleaned  from  analysis  to  date  helps  to  answer  the  research  questions.  It  is  important  to 
note  that  while  there  are  some  very  useful  insights  in  these  areas,  there  is  considerably  more  work  to  be 
done  before  the  questions  can  be  fully  answered. 


7.1  Question  1  -  What  Are  the  Characteristics  of  Systems  Engineers? 

Ideally,  to  answer  this  question  the  Helix  team  will  provide  a  landscape  view  of  the  systems  engineering 
population  and  be  able  to  draw  conclusions  about  commonalities.  However,  as  discussed  above,  it  has 
been  surprisingly  difficult  to  identify  systems  engineers  at  the  population  level,  so  at  the  time  of  this 
report,  the  team  must  rely  on  information  about  the  individuals  who  were  interviewed. 

The  team  does  have  early  insights  about  the  most  important  characteristics  and  competencies  of 
systems  engineers  (as  discussed  in  Section  6  above).  Perhaps  the  most  important  take  away  here  is  that 
there  is  a  mix  of  personal  characteristics  and  technical  competencies  that  seem  to  be  important  for 
creating  a  good  systems  engineer.  Technical  competencies  include  a  basic  understanding  of  engineering 
fundamentals,  the  ability  to  apply  these  fundamentals  to  some  depth  within  a  domain,  and  the  ability  to 
understand  other  engineering  disciplines  to  the  point  of  being  able  to  ask  insightful  questions.  These 
things  all  go  to  increasing  the  credibility  of  systems  engineers  as  leaders,  but  are  effectively  considered 
baseline  requirements.  To  that  end,  characteristics  such  as  strong  leadership,  communications,  big 
picture  thinking,  curiosity,  attention  to  detail,  humility,  etc.  -  which  are  sometimes  conflicting  -  seem  to 
be  common  in  systems  engineers.  Finally,  there  are  some  systems  engineering-specific  competencies 
that  are  important,  which  include  facets  such  as  understanding  the  systems  engineering  process,  tracing 
between  requirements  and  testing,  and  understanding  how  the  components  integrate  into  the 
complete  system. 


7.2  Question  2  -  How  Effective  Are  Systems  Engineers  and  Why? 

The  question  of  effectiveness  of  systems  engineers  is,  it  turns  out,  not  straightforward.  It  is  a 
combination  of  following  processes  and  procedures  and  helping  to  identify  issues  in  real  time  -  which 
can  generally  be  assessed  -  and  of  predicting  the  long-range  impacts  of  today's  actions.  This  time  lag 
complicates  the  understanding  of  effectiveness  both  for  systems  engineers  themselves  and  for  their 
counterparts  in  project  or  program  management,  other  engineering  disciplines,  and  organizational 
management. 

Despite  this,  however,  the  team  has  been  able  to  identify  patterns  in  what  benefits  systems  engineers 
believe  they  provide  to  their  projects  or  programs.  These  include  translating  highly  technical  information 
from  SMEs  into  common  language  that  other  stakeholders  can  understand;  getting  real  requirements 
from  the  customer  (versus  wants,  implementation  constraints,  etc.);  providing  a  balance  of  project 
management  concerns  (cost/schedule)  with  technical  requirements;  seeing  relationships  between 
engineering  disciplines  and  asking  the  right  questions  to  balance  these  disciplines;  managing  emergence, 
i.e.,  managing  the  issues  that  arise  under  uncertainty;  and  projecting  into  the  future.  Reflexively,  then, 
effective  systems  engineers  are  engineers  who  can  do  each  of  these  things. 

The  team  has  also  gained  some  additional  insights  into  what  conditions  most  improve  systems 
engineers'  effectiveness.  Though  many  items  were  discussed,  the  two  most  common  responses  were 
that  a  strong  mentoring  relationship  and  a  diverse  experience  base  are  very  important  for  systems 
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engineers  to  become  effective.  Conversely,  there  were  many  common  responses  to  what  factors  inhibit 
the  effectiveness  of  systems  engineers.  For  example,  lack  of  clarity  regarding  the  definition  of  systems 
engineering  or  who  within  an  organization  actually  performs  systems  engineering  tasks  dilutes  the 
organizational  view  of  the  value  of  systems  engineers  and  systems  engineering.  When  this  is  not 
understood,  it  becomes  difficult  to  justify  SE-related  efforts  when  budgets  and  schedules  are 
compressed,  as  is  often  seen  in  today's  environment.  Another  often-cited  detractor  from  valuing 
systems  engineers  is  the  time  delay  associated  with  systems  engineering.  Both  the  up  front  delay  -  i.e. 
time  spent  understanding  the  problem  and  not  'producing'  -  and  the  time  lag  between  when  decisions 
are  made  and  when  the  results  of  those  decisions  are  manifest  were  cited  as  complications  in 
understanding  the  value  of  SE  for  people  outside  the  discipline. 


7.3  Question  3  -  What  Are  Employers  Doing  to  Improve  Their  Systems  Engineers’ 
Effectiveness? 

While  the  team  does  have  some  insight  into  initiatives  being  conducted  to  improve  systems  engineering, 
it  is  impossible  for  the  team  to  draw  any  firm  conclusions  at  this  time.  With  only  seven  organizations 
providing  data  to  date,  the  sample  size  is  too  small  to  have  much  validity  and  would  be  too  revealing 
about  the  individual  organizations  participating  in  the  Helix  project.  During  2014,  with  a  large  sample  of 
organizations,  the  team  will  be  able  to  report  findings  for  Question  3. 

However,  as  discussed  above  it  can  be  noted  that  all  participating  organizations  recognized  the  risks 
associated  with  retirement  of  the  senior  systems  engineers,  who  represent  a  significant  portion  of  the 
workforce  in  many  places.  Three  common  initiatives  for  mitigating  this  risk  were:  initiation  of  more 
purposeful  mentoring  organizations;  succession  planning  with  associated  apprenticeship;  and 
improvements  to  knowledge  management. 
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8.  Future  Work 


There  are  some  specific  content  areas  of  interest  to  the  team  going  forward.  For  example,  the  team  will 
continue  to  work  with  participating  organizations  to  gain  more  information  on  organizational  initiatives, 
population  statistics,  etc.  Also,  the  team  to  date  has  worked  primarily  with  systems  engineers  or  systems 
engineering  managers;  the  team  will  focus  on  getting  participation  from  more  individuals  outside  of 
systems  engineering  -  specialty  engineers,  program  managers,  etc.  -  to  help  validate  some  of  the 
findings.  This  will  be  particularly  helpful  for  findings  such  as  systems  engineers'  perceived  value  added;  it 
will  give  the  external  perspective  and  help  to  validate  whether  what  systems  engineers  believe  they 
contribute  is  the  same  as  what  other  individuals  value  in  systems  engineers. 

The  team  will  also  work  on  a  deeper  dive  on  some  of  the  current  findings.  For  example,  there  is  data  on 
the  importance  of  a  diverse  set  of  experiences  for  systems  engineer  effectiveness,  but  the  team  will 
spend  some  focused  time  with  interviewees  exploring  this  in  more  depth.  The  same  will  be  true  for 
several  of  the  other  findings  outlined  in  this  report.  Additionally,  the  team  will  focus  effort  on  the 
collection  of  demographics  and  examine  how  the  demographic  information  influences  the 
interpretation  of  the  findings. 

The  Flelix  team  is  building  a  theory  of  systems  engineers  and  will  continue  to  conduct  interviews  and 
collect  and  analyze  other  forms  of  data  in  order  to  refine  and  validate  this  theory.  The  team  expects  to 
release  several  public  reports  in  2014  on  its  findings.  The  first  of  these  reports  will  outline  the  theory  of 
systems  engineers.  Additionally,  the  Flelix  team  plans  to  conduct  workshops  in  2014  where  SMEs  have 
an  opportunity  to  review  that  theory  and  associated  findings  and  help  the  team  to  validate  and  refine 
their  conclusions. 

The  team  is  also  examining  the  possibility  of  incorporating  additional  data  sources  for  2014,  including 
information  from  professional  societies  and  data  from  organizations  outside  the  US  defense  community. 

In  addition  to  data  from  individuals  and  participating  organizations,  Flelix  expects  to  capture  some 
environmental  data  that  may  have  influenced  the  data  collection.  For  example,  lay-offs,  mergers  & 
acquisitions,  freeze  on  hiring,  constraints  on  education  and  training,  changes  in  organization  leadership, 
project  successes  or  failures,  support  for  participation  in  conferences  and  other  professional  events,  and 
other  such  factors  could  significantly  affect  the  responses  of  the  interviewees.  It  is  proposed  to  capture 
such  environmental  factors  to  provide  a  context  to  the  data  being  collected. 
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Glossary 


Acronyms  &  Abbreviations 


Acronym/Abbreviation 

DASD(SE) 

DIB 

DOD 

HAP 

INCOSE 

IR&D 

IRB 

NDA 

NDIA-SED 

POC 

QRC 

SERC 

SME 

SPRDE 

UARC 


Meaning 

US  Deputy  Assistant  Secretary  of  Defense  for  Systems  Engineering 
Defense  Industrial  Base 
US  Department  of  Defense 
Helix  Advisory  Panel 

International  Council  on  Systems  Engineering 
Internal  (or  Independent)  Research  &  Development 
Internal  Review  Board 
Non-Disclosure  Agreement 

National  Defense  Industrial  Association  -  Systems  Engineering  Division 

Point  of  Contact 

Quick  Reaction  Capability 

Systems  Engineering  Research  Center 

Subject  Matter  Expert 

Systems  Planning,  Research,  Development,  and  Engineering 
University-Affiliated  Research  Center 


Terms 

characteristic  -  a  feature  or  quality  belonging  to  a  systems  engineer  and  serving  to  identify  systems 
engineers  as  a  group;  it  is  not  about  what  the  systems  engineer  knows  or  can  do  (see 

competency). 

competency  -  Knowledge  of  and  skill  in  the  practices  required  for  successful  development  of  a  system. 
There  are  3  types  of  relevant  competencies  for  Helix: 

•  Technical  competency  -  competency  relevant  to  a  specific  technical  discipline  or 
domain,  e.g.,  electrical,  mechanical,  or  civil  engineering  disciplines  or  the 
telecommunications,  engines,  or  ship  domains 

•  Systems  engineering  competency  -  competency  relevant  to  the  process, 
methodologies,  tools,  or  concepts  of  systems  engineering 

•  Business  competency  -  knowledge  and  skill  in  navigating  the  specific  workings  of 
the  organization  in  which  one  works 

effectiveness  -  The  degree  to  which  systems  engineering  activities  have  a  positive  impact  on  the 
outcomes  for  a  system.  For  example,  an  individual  systems  engineer  is  effective  when  the 
outcomes  for  which  he/she  is  individually  responsible  are  achieved  as  a  result  of  the  systems 
engineering  activities  he  performs  and  an  organization's  systems  engineering  workforce  is 
effective  when  the  outcomes  for  which  they  are  collectively  responsible  are  achieved  as  a  result 
of  the  systems  engineering  activities  they  perform. 

experience  -  Participation  in  or  observation  of  activities  during  which  an  individual  is  afforded  the  ability 
to  learn. 

•  Professional  experience  is  direct  observation  of,  participation  in,  or  leadership  of 
activities  in  a  work  environment. 
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•  Academic  experience  includes  any  activities  that  occur  within  an  educational  setting 
and  generally  will  include  formal  classroom  activities  such  as  participation  in  a 
degree  program. 

•  Social  experience  is  any  life  activity  (outside  the  classroom  or  professional  setting) 
that  is  relevant  to  the  effectiveness  of  systems  engineers. 

Notes:  The  Helix  team  focuses  on  professional  and  academic  experience,  but  does  collect 
information  on  social  experience  when  provided.  In  general,  the  Helix  team  focuses  on  suites  of 
experiences  rather  than  a  single,  isolated  experience. 

systems  engineering  -  An  interdisciplinary  approach  and  means  to  enable  the  realization  of  successful 
systems.  It  focuses  on  defining  customer  needs  and  required  functionality  early  in  the 
development  cycle,  documenting  requirements,  then  proceeding  with  design  synthesis  and 
system  validation  while  considering  the  complete  problem.  (INCOSE  2012) 
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Appendix  A  -  Sample  Interview  Questions 


Some  typical  questions  that  have  been  posed  to  junior  systems  engineers,  reflecting  on  their  personal 
situations,  are: 

•  What  activities  do  you  perform  most  frequently? 

•  What  are  the  most  important  activities  that  you  perform? 

•  Are  you  spending  the  right  amount  of  time  on  the  most  important  activities? 

•  What  were  your  expectations  when  you  first  took  this  position?  Does  that  align  with  what  you 
do  now? 

•  How  well  does  your  management  understand  and  appreciate  what  you  do? 

•  How  common  is  it  for  people  to  perform  systems  engineering  activities  who  are  not  classified  as 
systems  engineers  and  vice-versa? 

•  What  are  the  most  important  personal  traits  that  make  you  an  effective  systems  engineer? 
Why? 

•  What  are  the  most  important  forces  that  increase  your  effectiveness  as  a  systems  engineer? 
Why? 

•  What  are  the  most  important  forces  that  inhibit  your  effectiveness  as  a  systems  engineer?  Why? 

•  How  could  you  become  a  more  effective  systems  engineer? 

•  How  is  your  performance  as  a  systems  engineer  evaluated  and  rewarded?  What  metrics  are 
used? 

•  What  personal  initiatives  have  you  been  taking  to  improve  your  own  effectiveness? 

•  Which  organizational  initiatives  in  the  last  five  years  have  been  helping  improve  your 
effectiveness? 

Similar  questions  are  posed  to  more  senior-level  interviewees,  and  these  individuals  are  sought  to 
represent  not  just  themselves  but  the  organization  as  a  whole.  Some  typical  organizaton-related 
questions  are: 

•  Is  there  a  gap  between  the  effectiveness  of  your  systems  engineering  workforce  and  your 
organizational  need? 

•  How  is  the  performance  of  your  systems  engineering  workforce  evaluated  and  rewarded?  What 
metrics  are  used?  Are  they  uniform  across  the  roles  and  levels  of  the  systems  engineering 
workforce? 

•  What  are  the  primary  risks  to  the  systems  engineering  workforce  in  the  next  five  years?  How  are 
these  risks  being  addressed? 

•  Which  organizational  initiatives  in  the  last  five  years  have  had  the  greatest  impact  on  the  forces 
that  improve  workforce  effectiveness?  How  do  you  know  this? 

•  How  did  your  workforce  respond  to  these  initiatives? 

•  Are  these  initiatives  adequate  to  close  the  gap  between  effectiveness  and  organizational  need? 
What  more  should  be  done? 
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